JACIII Vol.27 No.5 pp. 967-975
doi: 10.20965/jaciii.2023.p0967

Research Paper:

Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training

Yi Liu, Jianzhang Li, Dewen Cui, and Eri Sato-Shimokawara ORCID Icon

Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan

March 31, 2023
June 27, 2023
September 20, 2023
emotional states classification, physiological signals, self-training, reinforcement learning

Affective computing focuses on recognizing emotions using a combination of psychology, computer science, and biomedical engineering. With virtual reality (VR) becoming more widely accessible, affective computing has become increasingly important for supporting social interactions on online virtual platforms. However, accurately estimating a person’s emotional state in VR is challenging because it differs from real-world conditions, such as the unavailability of facial expressions. This research proposes a self-training method that uses unlabeled data and a reinforcement learning approach to select and label data more accurately. Experiments on a dataset of dialogues of VR players show that the proposed method achieved an accuracy of over 80% on dominance and arousal labels and outperformed previous techniques in the few-shot classification of emotions based on physiological signals.

Cite this article as:
Y. Liu, J. Li, D. Cui, and E. Sato-Shimokawara, “Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training,” J. Adv. Comput. Intell. Intell. Inform., Vol.27 No.5, pp. 967-975, 2023.
Data files:
  1. [1] Y. Ikeda, R. Horie, and M. Sugaya, “Estimating emotion with biological information for robot interaction,” Procedia Computer Science, Vol.112, pp. 1589-1600, 2017.
  2. [2] R. Berrios, “What is complex/emotional about emotional complexity?,” Frontiers in Psychology, Vol.10, Article No.1606, 2019.
  3. [3] J. Marín-Morales, J. L. Higuera-Trujillo, A. Greco, J. Guixeres, C. Llinares, E. P. Scilingo, M. Alcañiz, and G. Valenza, “Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors,” Scientific Reports, Vol.8, Article No.13657, 2018.
  4. [4] C. N. W. Geraets, S. K. Tuente, B. P. Lestestuiver, M. van Beilen, S. A. Nijman, J. B. C. Marsman, and W. Veling, “Virtual reality facial emotion recognition in social environments: An eye-tracking study,” Internet Interventions, Vol.25, Article No.100432, 2021.
  5. [5] M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov, “Self-Training: A Survey,” arXiv:2202.12040, 2022.
  6. [6] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. V. Le, “Rethinking pre-training and self-training,” Proc. of the 34th Int. Conf. on Neural Information Processing Systems (NIPS’20), Article No.323, pp. 3833-3845, 2020.
  7. [7] Y. Zou, Z. Yu, X. Liu, B. V. K. V. Kumar, and J. Wang, “Confidence regularized self-training,” Proc. of 2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 5981-5990, 2019.
  8. [8] C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “CReST: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” Proc. of 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10852-10861, 2021.
  9. [9] M. Fang, Y. Li, and T. Cohn, “Learning how to active learn: A deep reinforcement learning approach,” arXiv:1708.02383, 2017.
  10. [10] C. Chen, Y. Zhang, and Y. Gao, “Learning how to self-learn: Enhancing self-training using neural reinforcement learning,” Proc. of 2018 Int. Conf. on Asian Language Processing (IALP), pp. 25-30, 2018.
  11. [11] J. Wu, L. Li, and W. Y. Wang, “Reinforced co-training,” arXiv:1804.06035, 2018.
  12. [12] Z. Ye, Y. Geng, J. Chen, J. Chen, X. Xu, S. Zheng, F. Wang, J. Zhang, and H. Chen, “Zero-shot text classification via reinforced self-training,” Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3014-3024, 2020.
  13. [13] K. Liu, Y. Fu, P. Wang, L. Wu, R. Bo, and X. Li, “Automating feature subspace exploration via multi-agent reinforcement learning,” Proc. of the 25th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’19), pp. 207-215, 2019.
  14. [14] J. Akosa, “Predictive accuracy: A misleading performance measure for highly imbalanced data,” Proc. of the SAS Global Forum, Paper 942-2017, 2017.
  15. [15] W. E. Mehling, C. Price, J. J. Daubenmier, M. Acree, E. Bartmess, and A. Stewart, “The multidimensional assessment of interoceptive awareness (MAIA),” PLoS one, Vol.7, No.11, Article No.e48230, 2012.
  16. [16] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,” J. of Behavior Therapy and Experimental Psychiatry, Vol.25, No.1, pp. 49-59, 1994.
  17. [17] P. Ekman, “Expression and the nature of emotion,” K. R. Scherer and P. Ekman (Eds.), “Approaches to Emotion,” pp. 319-334, L. Erlbaum Associates, 1984.
  18. [18] M. N. Dar, A. Rahim, M. U. Akram, S. G. Khawaja, and A. Rahim, “YAAD: Young Adult’s Affective Data Using Wearable ECG and GSR Sensors,” 2022 2nd Int. Conf. on Digital Futures and Transformative Technologies (ICoDT2), 2022.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Sep. 29, 2023