single-jc.php

JACIII Vol.27 No.5 pp. 967-975
doi: 10.20965/jaciii.2023.p0967
(2023)

Research Paper:

Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training

Yi Liu, Jianzhang Li, Dewen Cui, and Eri Sato-Shimokawara ORCID Icon

Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan

Received:
March 31, 2023
Accepted:
June 27, 2023
Published:
September 20, 2023
Keywords:
emotional states classification, physiological signals, self-training, reinforcement learning
Abstract

Affective computing focuses on recognizing emotions using a combination of psychology, computer science, and biomedical engineering. With virtual reality (VR) becoming more widely accessible, affective computing has become increasingly important for supporting social interactions on online virtual platforms. However, accurately estimating a person’s emotional state in VR is challenging because it differs from real-world conditions, such as the unavailability of facial expressions. This research proposes a self-training method that uses unlabeled data and a reinforcement learning approach to select and label data more accurately. Experiments on a dataset of dialogues of VR players show that the proposed method achieved an accuracy of over 80% on dominance and arousal labels and outperformed previous techniques in the few-shot classification of emotions based on physiological signals.

Cite this article as:
Y. Liu, J. Li, D. Cui, and E. Sato-Shimokawara, “Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training,” J. Adv. Comput. Intell. Intell. Inform., Vol.27 No.5, pp. 967-975, 2023.
Data files:
References
  1. [1] Y. Ikeda, R. Horie, and M. Sugaya, “Estimating emotion with biological information for robot interaction,” Procedia Computer Science, Vol.112, pp. 1589-1600, 2017. https://doi.org/10.1016/j.procs.2017.08.198
  2. [2] R. Berrios, “What is complex/emotional about emotional complexity?,” Frontiers in Psychology, Vol.10, Article No.1606, 2019. https://doi.org/10.3389/fpsyg.2019.01606
  3. [3] J. Marín-Morales, J. L. Higuera-Trujillo, A. Greco, J. Guixeres, C. Llinares, E. P. Scilingo, M. Alcañiz, and G. Valenza, “Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors,” Scientific Reports, Vol.8, Article No.13657, 2018. https://doi.org/10.1038/s41598-018-32063-4
  4. [4] C. N. W. Geraets, S. K. Tuente, B. P. Lestestuiver, M. van Beilen, S. A. Nijman, J. B. C. Marsman, and W. Veling, “Virtual reality facial emotion recognition in social environments: An eye-tracking study,” Internet Interventions, Vol.25, Article No.100432, 2021. https://doi.org/10.1016/j.invent.2021.100432
  5. [5] M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov, “Self-Training: A Survey,” arXiv:2202.12040, 2022. https://doi.org/10.48550/arXiv.2202.12040
  6. [6] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. V. Le, “Rethinking pre-training and self-training,” Proc. of the 34th Int. Conf. on Neural Information Processing Systems (NIPS’20), Article No.323, pp. 3833-3845, 2020.
  7. [7] Y. Zou, Z. Yu, X. Liu, B. V. K. V. Kumar, and J. Wang, “Confidence regularized self-training,” Proc. of 2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 5981-5990, 2019. https://doi.org/10.1109/ICCV.2019.00608
  8. [8] C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “CReST: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” Proc. of 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10852-10861, 2021. https://doi.org/10.1109/CVPR46437.2021.01071
  9. [9] M. Fang, Y. Li, and T. Cohn, “Learning how to active learn: A deep reinforcement learning approach,” arXiv:1708.02383, 2017. https://doi.org/10.48550/arXiv.1708.02383
  10. [10] C. Chen, Y. Zhang, and Y. Gao, “Learning how to self-learn: Enhancing self-training using neural reinforcement learning,” Proc. of 2018 Int. Conf. on Asian Language Processing (IALP), pp. 25-30, 2018. https://doi.org/10.1109/IALP.2018.8629107
  11. [11] J. Wu, L. Li, and W. Y. Wang, “Reinforced co-training,” arXiv:1804.06035, 2018. https://doi.org/10.48550/arXiv.1804.06035
  12. [12] Z. Ye, Y. Geng, J. Chen, J. Chen, X. Xu, S. Zheng, F. Wang, J. Zhang, and H. Chen, “Zero-shot text classification via reinforced self-training,” Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3014-3024, 2020. http://doi.org/10.18653/v1/2020.acl-main.272
  13. [13] K. Liu, Y. Fu, P. Wang, L. Wu, R. Bo, and X. Li, “Automating feature subspace exploration via multi-agent reinforcement learning,” Proc. of the 25th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’19), pp. 207-215, 2019. https://doi.org/10.1145/3292500.3330868
  14. [14] J. Akosa, “Predictive accuracy: A misleading performance measure for highly imbalanced data,” Proc. of the SAS Global Forum, Paper 942-2017, 2017.
  15. [15] W. E. Mehling, C. Price, J. J. Daubenmier, M. Acree, E. Bartmess, and A. Stewart, “The multidimensional assessment of interoceptive awareness (MAIA),” PLoS one, Vol.7, No.11, Article No.e48230, 2012. https://doi.org/10.1371/journal.pone.0048230
  16. [16] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,” J. of Behavior Therapy and Experimental Psychiatry, Vol.25, No.1, pp. 49-59, 1994. https://doi.org/10.1016/0005-7916(94)90063-9
  17. [17] P. Ekman, “Expression and the nature of emotion,” K. R. Scherer and P. Ekman (Eds.), “Approaches to Emotion,” pp. 319-334, L. Erlbaum Associates, 1984.
  18. [18] M. N. Dar, A. Rahim, M. U. Akram, S. G. Khawaja, and A. Rahim, “YAAD: Young Adult’s Affective Data Using Wearable ECG and GSR Sensors,” 2022 2nd Int. Conf. on Digital Futures and Transformative Technologies (ICoDT2), 2022. https://doi.org/10.1109/ICoDT255437.2022.9787465

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Mar. 01, 2024