Research Paper:
Effect of Perspective and Visuo-Tactile Feedback in Virtual Reality-Based Posture Learning
Ryo Hanashima*,**, Takumi Tokuda*,***, Masaaki Mochimaru*,*** , and Junji Ohyama*,**,
*National Institute of Advanced Industrial Science and Technology (AIST)
Kashiwa 2nd Campus, The University of Tokyo, 6-2-3 Kashiwa, Chiba 277-0882, Japan
**University of Tsukuba
Tsukuba, Japan
***The University of Tokyo
Kashiwa, Japan
Corresponding author
Posture learning is required in rehabilitation and in sports such as yoga and martial arts. Virtual reality (VR) systems are being used to learn posture by superimposing the postures of the learner and instructor in cyberspace using avatars. In this study, we examined whether the presented perspective of the avatar (first-person vs. third-person perspective) and visuo-tactile feedback (tactile correct feedback + visual feedback vs. tactile incorrect feedback + visual feedback vs. visual only feedback) are effective for the posture learning. The results of an experiment (N = 24) suggested that use of the third-person perspective may result in accurate learning of the head position as compared with first-person perspective. Visuo-tactile feedback was found to improve the subjective rating on the ease of learning, while the presentation method in which tactile feedback is given when body position is correct was found to be more effective than tactile feedback given when body position is incorrect. The sense of agency was maintained at a high level under all conditions for perspective and visuo-tactile feedback and may have improved the learning accuracy of posture. The findings of this study are expected to contribute to the design of effective perspective and tactile presentation in VR-based motor learning.
- [1] R. A. Schmidt, T. D. Lee, T. D. Carolee, G. Wulf, and H. Zelaznik, “Motor Control and Learning (sixth edition),” Human Kinetics, Champaign, 2018.
- [2] G. Wulf, C. Shea, and R. Lewthwaite, “Motor skill learning and performance: A review of influential factors,” Med. Educ., Vol.44, No.1, pp. 75-84, 2010. https://doi.org/10.1111/j.1365-2923.2009.03421.x
- [3] R. Sigrist, G. Rauter, R. Riener, and P. Wolf, “Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review,” Psychon. Bull. Rev., Vol.20, No.1, pp. 21-53, 2013. https://doi.org/10.3758/s13423-012-0333-8
- [4] D. L. Neumann, R. L. Moffitt, P. R. Thomas, K. Loveday, D. P. Watling, C. L. Lombard, S. Antonova, and M. A. Tremeer, “A systematic review of the application of interactive virtual reality to sport,” Virtual Reality, Vol.22, No.3, pp. 183-198, 2018. https://doi.org/10.1007/s10055-017-0320-5
- [5] A. Asadzadeh, T. Samad-Soltani, Z. Salahzadeh, and P. Rezaei-Hachesu, “Effectiveness of virtual reality-based exercise therapy in rehabilitation: A scoping review,” Inf. Med. Unlocked, Vol.24, 100562, 2021. https://doi.org/10.1016/j.imu.2021.100562
- [6] S. K. Renganayagalu, S. C. Mallam, and S. Nazir, “Effectiveness of VR Head Mounted Displays in Professional Training: A Systematic Review,” Technology, Knowledge and Learning, Vol.26, pp. 999-1041, 2021. https://doi.org/10.1007/s10758-020-09489-9
- [7] M. K. Holden and E. Todorov, “Use of Virtual Environments in Motor Learning and Rehabilitation,” M. K. Stanny (Ed.), “Handbook of Virtual Environments: Design, Implementation, and Applications,” Lawrence Erlbaum Associates, pp. 1-35, 2002.
- [8] T. Shank, “Measuring Mental Representations,” G. Tenenbaum, R. C. Eklund, and A. Kamata (Eds.), “Measurement in sport and exercise psychology,” Human Kinetics, Champaign, 2012.
- [9] T. Schack and F. Mechsner, “Representation of motor skills in human long-term memory,” Neurosci. Lett., Vol.391, No.3, pp. 77-81, 2006. https://doi.org/10.1016/j.neulet.2005.10.009
- [10] T. N. Hoang, M. Reinoso, F. Vetere, and E. Tanin, “Onebody: Remote posture guidance system using first person view in virtual environment,” Proc. of the 9th Nordic Conf. on Human-Computer Interaction (NordiCHI ’16), pp. 23-27, 2016. https://doi.org/10.1109/VR.2003.1191125
- [11] D. S. Kaesler, R. B. Mellifont, P. D. Kelly, and D. R, Taaffe, “A novel balance exercise program for postural stability in older adults: A pilot study,” J. Bodyw. Mov. Ther., Vol.11, No.1, pp. 37-43, 2007. https://doi.org/10.1016/j.jbmt.2006.05.003
- [12] P. T. Chua, R. Crivella, B. Daly, N. Hu, R. Schaaf, D. Ventura, T. Camill, J. Hodgins, and R. Pausch, “Training for physical tasks in virtual environments: Tai Chi,” IEEE Virtual Reality, pp. 87-94, 2003. https://doi.org/10.1109/VR.2003.1191125
- [13] T. Kojima, A. Hiyama, T. Miura, and M. Hirose, “Training Archived Physical Skill through Immersive Virtual Environment,” S. Yamamoto (Ed.), “Human Interface and the Management of Information. Information and Knowledge in Applications and Services,” HIMI 2014. Lecture Notes in Computer Science, Vol.8522, pp. 51-58, 2014. https://doi.org/10.1007/978-3-319-07863-2_6
- [14] F. Hülsmann, C. Frank, I. Senna, M. O. Ernst, T. Schack, and M. Botsch, “Superimposed Skilled Performance in a Virtual Mirror Improves Motor Performance and Cognitive Representation of a Full Body Motor Action,” Front. Rob. AI, Vol.6, 43, 2019. https://doi.org/10.3389/frobt.2019.00043
- [15] T. Le Naour, L. Hamon, and J.-P. Bresciani, “Superimposing 3D Virtual Self + Expert Modeling for Motor Learning: Application to the Throw in American Football,” Front. ICT, Vol.6, 16, 2019. https://doi.org/10.3389/fict.2019.00016
- [16] T. Waltemate, I. Senna, F. Hülsmann, M. Rohde, S. Kopp, M. Ernst, and M. Botsch, “The impact of latency on perceptual judgments and motor performance in closed-loop interaction in virtual reality,” Proc. of the 22nd ACM Conf. on Virtual Reality Software and Technology (VRST ’16), pp. 27-35, 2016. https://doi.org/10.1145/2993369.2993381
- [17] S. Gallagher, “Philosophical conceptions of the self: implications for cognitive science,” Trends Cognit. Sci., Vol.4, No.1, pp. 14-21, 2000. https://doi.org/10.1016/S1364-6613(99)01417-5
- [18] M. Tsakiris, G. Prabhu, and P. Haggard, “Having a body versus moving your body: How agency structures body-ownership,” Conscious Cogn., Vol.15, No.2, pp. 423-432, 2006. https://doi.org/10.1016/j.concog.2005.09.004
- [19] O. Blanke and T. Metzinger, “Full-body illusions and minimal phenomenal selfhood,” Trends Cognit. Sci., Vol.13, No.1, pp. 7-13, 2009. https://doi.org/10.1016/j.tics.2008.10.003
- [20] K. Kilteni, R. Groten, and M. Slater, “The Sense of embodiment in virtual reality,” Presence (Camb), Vol.22, No.1, pp. 373-387, 2013. https://doi.org/10.1162/PRES_a_00124
- [21] T. Asai, “Feedback control of one’s own action: Self-other sensory attribution in motor control,” Conscious Cogn., Vol.38, pp. 118-129, 2015. https://doi.org/10.1016/j.concog.2015.11.002
- [22] D. Burin, K. Kilteni, M. Rabuffetti, and L. Pia, “Body ownership increases the interference between observed and executed movements,” PLoS One, Vol.14, No.1, e0209899, 2019. https://doi.org/10.1371/journal.pone.0209899
- [23] K. Grechuta, J. Guga, G. Maffei, B. R. Balleater, and P. F. M. J. Verschure, “Visuotactile integration modulates motor performance in a perceptual decision-making task,” Sci. Rep., Vol.7, 3333, 2017. https://doi.org/10.1038/s41598-017-03488-0
- [24] R. Newport, R. Pearce, and C. Preston, “Fake hands in action: Embodiment and control of supernumerary limbs,” Exp. Brain Res., Vol.204, No.3, pp. 385-395, 2010. https://doi.org/10.1007/s00221-009-2104-y
- [25] R. Zopf, S. Truong, M. Finkbeiner, J. Friedman, and M. A. Williams, “Viewing and feeling touch modulates hand position for reaching,” Neuropsychologia, Vol.49, No.5, pp. 1287-1293, 2011. https://doi.org/10.1016/j.neuropsychologia.2011.02.012
- [26] M. P. M. Kammers, F. de Vignemont, L. Verhagen, and H. C. Dijkerman, “The rubber hand illusion in action,” Neuropsychologia, Vol.47, pp. 204-211, 2009. https://doi.org/10.1016/j.neuropsychologia.2008.07.028
- [27] K. Matsumiya, “Awareness of voluntary action, rather than body ownership, improves motor control,” Sci. Rep., Vol.11, 418, 2021. https://doi.org/10.1038/s41598-020-79910-x
- [28] I. A. Odermatt, K. A. Buetler, N. Wenk, Ö. Özen, J. Penalver-Andres, T. Nef, F. W. Mast, and L. Marchal-Crespo, “Congruency of Information Rather Than Body Ownership Enhances Motor Performance in Highly Embodied Virtual Reality,” Front. Neurosci., Vol.15, 678909, 2021. https://doi.org/10.3389/fnins.2021.678909
- [29] A. T. Reader, V. S. Trifonova, and H. H. Ehrsson, “Little evidence for an effect of the rubber hand illusion on basic movement,” Eur. J. Neurosci., Vol.7, No.54, pp. 6463-6486, 2021. https://doi.org/10.1111/ejn.15444
- [30] S. Malpica, A. Serrano, and D. Gutierrez, “Auditory stimuli degrade visual performance in virtual reality,” Sci. Rep., Vol.10, 12363, 2020. https://doi.org/10.1038/s41598-020-69135-3
- [31] R. Sigrist, G. Rauter, L. Marchal-Crespo, R. Riener, and P. Wolf, “Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning,” Exp. Brain. Res., Vol.233, No.3, pp. 909-925, 2015. https://doi.org/10.1007/s00221-014-4167-7
- [32] J. Ohyama, “Xperigrapher: Social-Lab Experimental Platform to Evaluate Experience in Cyber Physical Society,” Trans. Japan. Soc. Med. Biol. Eng., Vol.59, pp. 811-813, 2021. https://doi.org/10.11239/jsmbe.Annual59.811
- [33] T. Maeno, “Structure and Function of Finger Pad and Tactile Receptors,” J. of the Robotics Society of Japan, Vol.18, No.6, pp. 772-775, 2000 (in Japanese). https://doi.org/10.7210/jrsj.18.772
- [34] T. C. Peck and M. Gonzalez-Franco, “Avatar Embodiment. A Standardized Questionnaire,” Front. Virtual Reality, Vol.1, 575943, 2021. https://doi.org/10.3389/frvir.2020.575943
- [35] H. Shimizu, “An introduction to the statistical free software HAD: Suggestions to improve teaching, learning and practice data analysis,” J. of Media, Information and Communication, Vol.1, pp. 59-73, 2016 (in Japanese).
- [36] T. Hashimoto and A. Iriki, “Somatosensation,” Brain Science Dictionary, 2012 (in Japanese). https://bsd.neuroinf.jp/wiki/%E4%BD%93%E6%80%A7%E6%84%9F%E8%A6%9A [Accessed September 6, 2022].
- [37] G. Gorisse, O. Christmann, E. A. Amato, and S. Richir, “First- and Third-Person Perspectives in Immersive Virtual Environments: Presence and Performance Analysis of Embodied Users,” Front. Rob. AI, Vol.4, 33, 2017. https://doi.org/10.3389/frobt.2017.00033
- [38] F. Argelaguet, L. Hoyet, M. Trico, and A. Lécuyer, “The role of interaction in virtual embodiment: Effects of the virtual hand representation,” 2016 IEEE Virtual Reality (VR), pp. 3-10, 2016. https://doi.org/10.1109/VR.2016.7504682
- [39] R. Hanashima and J. Ohyama, “Experimental psychology study using virtual reality: The effects of avatar size on somatosensory transference,” IPSJ Special Interest Group on Entertainment Computing, Vol.54, No.11, pp. 1-2, 2019 (in Japanese).
- [40] W. Choi, L. Li, S. Satoh, and K. Hachimura, “Multisensory Integration in the Virtual Hand Illusion with Active Movement,“ Biomed Res. Int., Vol.2016, 8163098, 2016. https://doi.org/10.1155/2016/8163098
- [41] C. D. Frith, S. J. Blakemore, and D. W. Wolpert, “Abnormalities in the awareness and control of action,” Philos. Trans. R. Soc. B: Biol. Sci., Vol.355, pp. 1771-1788, 2000. https://doi.org/10.1098/rstb.2000.0734
- [42] C. D. Frith, S. Blakemore, and D. W. Wolpert, “Explaining the symptoms of schizophrenia: abnormalities in the awareness of action,” Brain Res. Rev., Vol.31, No.2-3, pp. 357-363, 2000. https://doi.org/10.1016/s0165-0173(99)00052-1
- [43] R. Hanashima and J. Ohyama, “How to Elicit Ownership and Agency for an Avatar Presented in the Third-Person Perspective: The Effect of Visuo-Motor and Tactile Feedback,” S. Yamamoto and H. Mori (Eds.), “Human Interface and the Management of Information: Applications in Complex Technological Environments,” HCII 2022. Lecture Notes in Computer Science, Vol.13306, pp. 111-130, 2022. https://doi.org/10.1007/978-3-031-06509-5_9
- [44] M. D’Angelo, G. di Pellegrino, S. Seriani, P. Gallina, and F. Frassinetti, “The sense of agency shapes body schema and peripersonal space,” Sci. Rep., Vol.8, 13847, 2018. https://doi.org/10.1038/s41598-018-32238-z
- [45] C. J. Winstein and R. A. Schmidt, “Reduced frequency of knowledge of results enhances motor skill learning,” J. Exp. Psychol.: Learn. Mem. Cogn., Vol.16, No.4, pp. 677-91, 1990. https://doi.org/10.1037/0278-7393.16.4.677
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.