Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training

Yi Liu; Jianzhang Li; Dewen Cui; Eri Sato-Shimokawara

doi:10.20965/jaciii.2023.p0967

single-jc.php

« previous

JACIII Vol.27 No.5 pp. 967-975

doi: 10.20965/jaciii.2023.p0967

(2023)

Research Paper:

Views over last 60 days: 208

Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training

Yi Liu, Jianzhang Li, Dewen Cui, and Eri Sato-Shimokawara

Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan

Received:

March 31, 2023

Accepted:

June 27, 2023

Published:

September 20, 2023

Keywords:

emotional states classification, physiological signals, self-training, reinforcement learning

Abstract

Affective computing focuses on recognizing emotions using a combination of psychology, computer science, and biomedical engineering. With virtual reality (VR) becoming more widely accessible, affective computing has become increasingly important for supporting social interactions on online virtual platforms. However, accurately estimating a person’s emotional state in VR is challenging because it differs from real-world conditions, such as the unavailability of facial expressions. This research proposes a self-training method that uses unlabeled data and a reinforcement learning approach to select and label data more accurately. Experiments on a dataset of dialogues of VR players show that the proposed method achieved an accuracy of over 80% on dominance and arousal labels and outperformed previous techniques in the few-shot classification of emotions based on physiological signals.

Cite this article as:

Y. Liu, J. Li, D. Cui, and E. Sato-Shimokawara, “Multi-Modal Emotion Classification in Virtual Reality Using Reinforced Self-Training,” J. Adv. Comput. Intell. Intell. Inform., Vol.27 No.5, pp. 967-975, 2023.

Data files:

References

[1] Y. Ikeda, R. Horie, and M. Sugaya, “Estimating emotion with biological information for robot interaction,” Procedia Computer Science, Vol.112, pp. 1589-1600, 2017. https://doi.org/10.1016/j.procs.2017.08.198
[2] R. Berrios, “What is complex/emotional about emotional complexity?,” Frontiers in Psychology, Vol.10, Article No.1606, 2019. https://doi.org/10.3389/fpsyg.2019.01606
[3] J. Marín-Morales, J. L. Higuera-Trujillo, A. Greco, J. Guixeres, C. Llinares, E. P. Scilingo, M. Alcañiz, and G. Valenza, “Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors,” Scientific Reports, Vol.8, Article No.13657, 2018. https://doi.org/10.1038/s41598-018-32063-4
[4] C. N. W. Geraets, S. K. Tuente, B. P. Lestestuiver, M. van Beilen, S. A. Nijman, J. B. C. Marsman, and W. Veling, “Virtual reality facial emotion recognition in social environments: An eye-tracking study,” Internet Interventions, Vol.25, Article No.100432, 2021. https://doi.org/10.1016/j.invent.2021.100432
[5] M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov, “Self-Training: A Survey,” arXiv:2202.12040, 2022. https://doi.org/10.48550/arXiv.2202.12040
[6] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. V. Le, “Rethinking pre-training and self-training,” Proc. of the 34th Int. Conf. on Neural Information Processing Systems (NIPS’20), Article No.323, pp. 3833-3845, 2020.
[7] Y. Zou, Z. Yu, X. Liu, B. V. K. V. Kumar, and J. Wang, “Confidence regularized self-training,” Proc. of 2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 5981-5990, 2019. https://doi.org/10.1109/ICCV.2019.00608
[8] C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “CReST: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” Proc. of 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10852-10861, 2021. https://doi.org/10.1109/CVPR46437.2021.01071
[9] M. Fang, Y. Li, and T. Cohn, “Learning how to active learn: A deep reinforcement learning approach,” arXiv:1708.02383, 2017. https://doi.org/10.48550/arXiv.1708.02383
[10] C. Chen, Y. Zhang, and Y. Gao, “Learning how to self-learn: Enhancing self-training using neural reinforcement learning,” Proc. of 2018 Int. Conf. on Asian Language Processing (IALP), pp. 25-30, 2018. https://doi.org/10.1109/IALP.2018.8629107
[11] J. Wu, L. Li, and W. Y. Wang, “Reinforced co-training,” arXiv:1804.06035, 2018. https://doi.org/10.48550/arXiv.1804.06035
[12] Z. Ye, Y. Geng, J. Chen, J. Chen, X. Xu, S. Zheng, F. Wang, J. Zhang, and H. Chen, “Zero-shot text classification via reinforced self-training,” Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3014-3024, 2020. http://doi.org/10.18653/v1/2020.acl-main.272
[13] K. Liu, Y. Fu, P. Wang, L. Wu, R. Bo, and X. Li, “Automating feature subspace exploration via multi-agent reinforcement learning,” Proc. of the 25th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’19), pp. 207-215, 2019. https://doi.org/10.1145/3292500.3330868
[14] J. Akosa, “Predictive accuracy: A misleading performance measure for highly imbalanced data,” Proc. of the SAS Global Forum, Paper 942-2017, 2017.
[15] W. E. Mehling, C. Price, J. J. Daubenmier, M. Acree, E. Bartmess, and A. Stewart, “The multidimensional assessment of interoceptive awareness (MAIA),” PLoS one, Vol.7, No.11, Article No.e48230, 2012. https://doi.org/10.1371/journal.pone.0048230
[16] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,” J. of Behavior Therapy and Experimental Psychiatry, Vol.25, No.1, pp. 49-59, 1994. https://doi.org/10.1016/0005-7916(94)90063-9
[17] P. Ekman, “Expression and the nature of emotion,” K. R. Scherer and P. Ekman (Eds.), “Approaches to Emotion,” pp. 319-334, L. Erlbaum Associates, 1984.
[18] M. N. Dar, A. Rahim, M. U. Akram, S. G. Khawaja, and A. Rahim, “YAAD: Young Adult’s Affective Data Using Wearable ECG and GSR Sensors,” 2022 2nd Int. Conf. on Digital Futures and Transformative Technologies (ICoDT2), 2022. https://doi.org/10.1109/ICoDT255437.2022.9787465

This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

[1] [1] Y. Ikeda, R. Horie, and M. Sugaya, “Estimating emotion with biological information for robot interaction,” Procedia Computer Science, Vol.112, pp. 1589-1600, 2017. https://doi.org/10.1016/j.procs.2017.08.198

[2] [2] R. Berrios, “What is complex/emotional about emotional complexity?,” Frontiers in Psychology, Vol.10, Article No.1606, 2019. https://doi.org/10.3389/fpsyg.2019.01606

[3] [3] J. Marín-Morales, J. L. Higuera-Trujillo, A. Greco, J. Guixeres, C. Llinares, E. P. Scilingo, M. Alcañiz, and G. Valenza, “Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors,” Scientific Reports, Vol.8, Article No.13657, 2018. https://doi.org/10.1038/s41598-018-32063-4

[4] [4] C. N. W. Geraets, S. K. Tuente, B. P. Lestestuiver, M. van Beilen, S. A. Nijman, J. B. C. Marsman, and W. Veling, “Virtual reality facial emotion recognition in social environments: An eye-tracking study,” Internet Interventions, Vol.25, Article No.100432, 2021. https://doi.org/10.1016/j.invent.2021.100432

[5] [5] M.-R. Amini, V. Feofanov, L. Pauletto, E. Devijver, and Y. Maximov, “Self-Training: A Survey,” arXiv:2202.12040, 2022. https://doi.org/10.48550/arXiv.2202.12040

[6] [6] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. V. Le, “Rethinking pre-training and self-training,” Proc. of the 34th Int. Conf. on Neural Information Processing Systems (NIPS’20), Article No.323, pp. 3833-3845, 2020.

[7] [7] Y. Zou, Z. Yu, X. Liu, B. V. K. V. Kumar, and J. Wang, “Confidence regularized self-training,” Proc. of 2019 IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 5981-5990, 2019. https://doi.org/10.1109/ICCV.2019.00608

[8] [8] C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “CReST: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” Proc. of 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10852-10861, 2021. https://doi.org/10.1109/CVPR46437.2021.01071

[9] [9] M. Fang, Y. Li, and T. Cohn, “Learning how to active learn: A deep reinforcement learning approach,” arXiv:1708.02383, 2017. https://doi.org/10.48550/arXiv.1708.02383

[10] [10] C. Chen, Y. Zhang, and Y. Gao, “Learning how to self-learn: Enhancing self-training using neural reinforcement learning,” Proc. of 2018 Int. Conf. on Asian Language Processing (IALP), pp. 25-30, 2018. https://doi.org/10.1109/IALP.2018.8629107

[11] [11] J. Wu, L. Li, and W. Y. Wang, “Reinforced co-training,” arXiv:1804.06035, 2018. https://doi.org/10.48550/arXiv.1804.06035

[12] [12] Z. Ye, Y. Geng, J. Chen, J. Chen, X. Xu, S. Zheng, F. Wang, J. Zhang, and H. Chen, “Zero-shot text classification via reinforced self-training,” Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3014-3024, 2020. http://doi.org/10.18653/v1/2020.acl-main.272

[13] [13] K. Liu, Y. Fu, P. Wang, L. Wu, R. Bo, and X. Li, “Automating feature subspace exploration via multi-agent reinforcement learning,” Proc. of the 25th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’19), pp. 207-215, 2019. https://doi.org/10.1145/3292500.3330868

[14] [14] J. Akosa, “Predictive accuracy: A misleading performance measure for highly imbalanced data,” Proc. of the SAS Global Forum, Paper 942-2017, 2017.

[15] [15] W. E. Mehling, C. Price, J. J. Daubenmier, M. Acree, E. Bartmess, and A. Stewart, “The multidimensional assessment of interoceptive awareness (MAIA),” PLoS one, Vol.7, No.11, Article No.e48230, 2012. https://doi.org/10.1371/journal.pone.0048230

[16] [16] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,” J. of Behavior Therapy and Experimental Psychiatry, Vol.25, No.1, pp. 49-59, 1994. https://doi.org/10.1016/0005-7916(94)90063-9

[17] [17] P. Ekman, “Expression and the nature of emotion,” K. R. Scherer and P. Ekman (Eds.), “Approaches to Emotion,” pp. 319-334, L. Erlbaum Associates, 1984.

[18] [18] M. N. Dar, A. Rahim, M. U. Akram, S. G. Khawaja, and A. Rahim, “YAAD: Young Adult’s Affective Data Using Wearable ECG and GSR Sensors,” 2022 2nd Int. Conf. on Digital Futures and Transformative Technologies (ICoDT2), 2022. https://doi.org/10.1109/ICoDT255437.2022.9787465