JRM Vol.35 No.5 pp. 1331-1339
doi: 10.20965/jrm.2023.p1331


Speech-Driven Avatar Robot System with Changing Complexion for the Visualization of an Interactive Atmosphere

Yoshihiro Sejima* ORCID Icon, Liheng Yang**, Saki Inagaki**, and Daiki Morita**

*Faculty of Informatics, Kansai University
2-1-1 Ryozenji-cho, Takatsuki, Osaka 569-1095, Japan

**Graduate School of Informatics, Kansai University
2-1-1 Ryozenji-cho, Takatsuki, Osaka 569-1095, Japan

January 24, 2023
July 3, 2023
October 20, 2023
nonverbal communication, human-robot interaction, interactive atmosphere, affective display

Smooth interactions between talkers can be realized by transmitting and receiving mutual video images and voices in remote communication. However, in such remote communication, it is difficult to generate a sense of unity and an interactive atmosphere because humans recognize screens as a boundary of the physical space. Therefore, it is essential to develop a communication system that can generate and share an interactive atmosphere and interaction-activated communication even if talkers are in remote places. In this study, we developed a speech-driven avatar robot system incorporating an estimation model that simulates the degree of activated communication based on the talker’s speech. The developed avatar robot system can visualize an interactive atmosphere while changing the complexion based on an estimated value. The effectiveness of the developed system was demonstrated by means of sensory evaluations.

Talker can grasp the atmosphere from the complexion of the avatar robot during remote communication

Talker can grasp the atmosphere from the complexion of the avatar robot during remote communication

Cite this article as:
Y. Sejima, L. Yang, S. Inagaki, and D. Morita, “Speech-Driven Avatar Robot System with Changing Complexion for the Visualization of an Interactive Atmosphere,” J. Robot. Mechatron., Vol.35 No.5, pp. 1331-1339, 2023.
Data files:
  1. [1] S. Dash, S. Samadder, A. Srivastava, R. Meena, and P. Ranjan, “Review of online teaching platforms in the current period of COVID-19 pandemic,” Indian J. of Surgery, Vol.84, Suppl.1, pp. 12-17, 2022.
  2. [2] T. Watanabe, “Human-entrained embodied interaction and communication technology for human-connected IoT design,” J. of Advanced Mechanical Design, Systems, and Manufacturing, Vol.14, No.2, Article No.19-00322, 2020.
  3. [3] T. Watanabe, “Human-entrained embodied interaction and communication technology,” S. Fukuda (Ed.), “Emotional Engineering: Service Development,” pp. 161-177, Springer, 2011.
  4. [4] Y. Sejima, T. Watanabe, and M. Jindai, “Estimation model of interaction-activated communication based on the heat conduction equation,” J. of Advanced Mechanical Design, Systems, and Manufacturing, Vol.10, No.9, Article No.15-00548, 2016.
  5. [5] F. P. Incropera and D. P. DeWitt, “Introduction to Heat Transfer,” 4th Edition, John Wiley & Sons, 2002.
  6. [6] Y. Sejima, Y. Ishii, and T. Watanabe, “An embodied communication system with avatar-shadow’s color expressions based on an interaction-activated communication model in voice communication,” Trans. of the JSME, Vol.85, No.873, Article No.18-00074, 2019 (in Japanese).
  7. [7] K. Takeuchi, Y. Yamazaki, and K. Yoshifuji, “Avatar work: Telework for disabled people unable to go outside by using avatar robots,” Companion of the 2020 ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI’20), pp. 53-60, 2020.
  8. [8] R. Yamazaki, H. Kase, S. Nishio, and H. Ishiguro, “Anxiety reduction through close communication with robotic media in dementia patients and healthy older adults,” J. of Robot. Mechatron., Vol.32, No.1, pp. 32-42, 2020.
  9. [9] T. Ito and T. Watanabe, “Natural involvement to video conference through ARM-COMS,” S. Yamamoto and H. Mori (Eds.), “Human Interface and the Management of Information: Applications in Complex Technological Environments,” pp. 238-246, Springer, 2022.
  10. [10] S. Yamamoto, J. Woo, W. H. Chin, K. Matsumura, and N. Kubota, “Interactive information support by robot partners based on informationally structured space,” J. Robot. Mechatron., Vol.32, No.1, pp. 236-243, 2020.
  11. [11] M. Shiomi, H. Sumioka, and H. Ishiguro, “Survey of social touch interaction between humans and robots,” J. Robot. Mechatron., Vol.32, No.1, pp. 128-135, 2020.
  12. [12] Y. Seong, “Design research of wearable soft avatar robot for interactive social presence,” J. Robot. Mechatron., Vol.34, No.2, pp. 325-327, 2022.
  13. [13] Y. Inoue and M. Kitazaki, “Virtual mirror and beyond: The psychological basis for avatar embodiment via a mirror,” J. Robot. Mechatron., Vol.33, No.5, pp. 1004-1012, 2021.
  14. [14] A. M. Soccini, A. Clocchiatti, and T. Inamura, “Effects of frequent changes in extended self-avatar movements on adaptation performance,” J. Robot. Mechatron., Vol.34, No.4, pp. 756-766, 2021.
  15. [15] K. Higashi, N. Isoyama, N. Sakata, and K. Kiyokawa, “Manipulating sense of participation in multipartite conversations by manipulating head attitude and gaze direction,” J. Robot. Mechatron., Vol.33, No.5, pp. 1013-1028, 2021.
  16. [16] J. N. Bailenson, “Nonverbal overload: A theoretical argument for the causes of Zoom fatigue,” Technology, Mind, and Behavior, Vol.2, No.1, 2021.
  17. [17] T. Obo and K. Takizawa, “Analysis of timing and effect of visual cue on turn-taking in human-robot interaction,” J. Robot. Mechatron., Vol.34, No.1, pp. 55-63, 2022.
  18. [18] L. Yang, S. Inagaki, D. Morita, and Y. Sejima, “Development of an avatar-robot with interactive atmosphere in online communication,” The Proc. of 2022 JSME Annual Conf. on Robotics and Mechatronics (Robomec), Session ID 2A1-H07, 2022 (in Japanese).
  19. [19] V. M. Reid, K. Dunn, R. J. Young, J. Amu, T. Donovan, and N. Reissland, “The human fetus preferentially engages with face-like visual stimuli,” Current Biology, Vol.27, No.12, pp.1825-1828, 2017.
  20. [20] Y. Sejima, S. Egawa, R. Maeda, Y. Sato, and T. Watanabe, “A speech-driven pupil response robot synchronized with burst-pause of utterance,” 2017 26th IEEE Int. Symp. on Robot and Human Interactive Communication (RO-MAN), pp. 437-442, 2017.
  21. [21] T. Watanabe, M. Ogikubo, and Y. Ishii, “Visualization of respiration in the embodied virtual communication system and its evaluation,” Int. J. of Human-Computer Interaction, Vol.17, No.1, pp. 89-102, 2004.
  22. [22] I. Inagaki, D. Morita, and Y. Sejima, “Development of a communication system that expresses interactive atmosphere with changing the background color using interaction-activated communication model,” Proc. of the Symp. on Human Interface 2022, pp. 533-537, 2022 (in Japanese).
  23. [23] Y. Sejima, Y. Sato, T. Watanabe, and M. Jindai, “Speech-driven embodied entrainment character system with pupillary response,” Mechanical Engineering J., Vol.3, No.4, Article No.15-00314, 2016.
  24. [24] R. D. Luce, “Individual choice behaviour: A theoretical analysis,” J. Wiley, 1959.
  25. [25] P. Ekman, R. W. Levenson, and W. V. Friesen, “Autonomic nervous system activity distinguishes among emotions,” Science, Vol.221, No.4616, pp. 1208-1210, 1983.
  26. [26] T. Yamada and T. Watanabe, “An average facial color image avatar system for the analysis by synthesis of affect display by dynamic facial color and expression,” Trans. of the Japan Society of Mechanical Engineers, Series C, Vol.78, No.791, pp. 2526-2535, 2012 (in Japanese).
  27. [27] D. McDuff and E. M. Nowara, “‘Warm bodies’: A post-processing technique for animating dynamic blood flow on photos and avatars,” Proc. of the 2021 Conf. on Human Factors in Computing Systems (CHI’21), Article No.579, 2021.
  28. [28] M. Hori, Y. Tsuruda, H. Yoshimura, and Y. Iwai, “Expression transmission using exaggerated animation for Elfoid,” Frontiers in Psychology, Vol.6, Article No.1219, 2015.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Nov. 24, 2023