This paper presents a virtual reality and digital twin-based training system designed to improve human-robot collaboration in retail store environments, particularly under disaster scenarios. This system enables dynamic role adaptation between humans and AI-controlled avatars or robots, facilitating diverse collaborative configurations. In a virtual retail environment replicating post-disaster conditions, human subjects—paired either with AI or another human participant—engage in collaborative object-retrieval tasks, distinguishing between safe and hazardous items. Experimental results indicate that human-human collaborations outperform human-AI collaborations in both task efficiency and safety. Participants exhibited improved movement efficiency and higher accuracy in retrieving safe items when paired with another human. These findings suggest that human-robot interaction training can benefit from human-human collaboration configurations for skill enhancement. This system also demonstrates potential for broader applications in simulating complex hazardous environments where real-world training is challenging.

1. [1] M. Walker, T. Phung, T. Chakraborti, T. Williams, and D. Szafir, “Virtual, augmented, and mixed reality for human-robot interaction: A survey and virtual design element taxonomy,” ACM Trans. Hum. Robot Interact., Vol.12, No.4, Article No.43, 2023. https://doi.org/10.1145/3597623
2. [2] R. Suzuki, A. Karim, T. Xia, H. Hedayati, and N. Marquardt, “Augmented reality and robotics: A survey and taxonomy for AR-enhanced human-robot interaction and robotic interfaces,” Proc. of the 2022 CHI Conf. on Human Factors in Computing Systems, Article No.553, 2022. https://doi.org/10.1145/3491102.3517719
3. [3] N. Xi and J. Hamari, “Shopping in virtual reality: A literature review and future agenda,” J. Bus. Res., Vol.134, pp. 37-58, 2021. https://doi.org/10.1016/j.jbusres.2021.04.075
4. [4] V. Villani, F. Pini, F. Leali, and C. Secchi, “Survey on human-robot collaboration in industrial settings: Safety, intuitive interfaces and applications,” Mechatronics, Vol.55, pp. 248-266, 2018. https://doi.org/10.1016/j.mechatronics.2018.02.009
5. [5] E. Matsas and G.-C. Vosniakos, “Design of a virtual reality training system for human-robot collaboration in manufacturing tasks,” Int. J. Interact. Des. Manuf., Vol.11, No.2, pp. 139-153, 2017. https://doi.org/10.1007/s12008-015-0259-2
6. [6] Z. Liao and Y. Cai, “AR-enhanced digital twin for human-robot interaction in manufacturing systems,” Energy Ecol. Environ., Vol.9, pp. 530-548, 2024. https://doi.org/10.1007/s40974-024-00327-7
7. [7] B. Xie, H. Liu, R. Alghofaili, Y. Zhang, Y. Jiang, F. D. Lobo, C. Li, W. Li, H. Huang, M. Akdere, C. Mousas, and L.-F. Yu, “A review on virtual reality skill training applications,” Front. Virtual Real., Vol.2, 2021. https://doi.org/10.3389/frvir.2021.645153
8. [8] T. Kanda, M. Shiomi, Z. Miyashita, H. Ishiguro, and N. Hagita, “A communication robot in a shopping mall,” IEEE Trans. Rob., Vol.26, Issue 5, pp. 897-913, 2010. https://doi.org/10.1109/TRO.2010.2062550
9. [9] M. Veloso, J. Biswas, B. Coltin, and S. Rosenthal, “CoBots: Robust symbiotic autonomous mobile service robots,” Proc. of the 24th Int. Conf. on Artificial Intelligence (IJCAI’15), pp. 4423-4429, 2015.
10. [10] M. A. Goodrich and A. C. Schultz, “Human-robot interaction: A survey,” Now Foundations and Trends, 2008. https://doi.org/10.1561/1100000005
11. [11] K. Wan, C. Li, F.-S. Lo, and P. Zheng, “A virtual reality-based immersive teleoperation system for remote human-robot collaborative manufacturing,” Manuf. Lett., Vol.41, No.Supplement, pp. 43-50, 2024. https://doi.org/10.1016/j.mfglet.2024.09.008
12. [12] L. Pérez, E. Diez, R. Usamentiaga, and D. F. García, “Industrial robot control and operator training using virtual reality interfaces,” Comput. Ind., Vol.109, pp. 114-120, 2019. https://doi.org/10.1016/j.compind.2019.05.001
13. [13] P. Adami, P. B. Rodrigues, P. J. Woods, B. Becerik-Gerber, L. Soibelman, Y. Copur-Gencturk, and G. Lucas, “Impact of VR-based training on human-robot interaction for remote operating construction robots,” J. Comput. Civ. Eng., Vol.36, Issue 3, Article No.04022006, 2022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001016
14. [14] A. Onososen, I. Musonda, C. Dzuwa, and R. Molusiwa, “Design and development of virtual reality environment for human-robot interaction on construction site,” C. Park et al. (Eds.), “Digitalization in Construction,” pp. 311-321, Routledge, 2023.
15. [15] Y. Zhao, L. Masuda, L. Loke, and D. Reinhardt, “Designing a Dynamically Configurable Digital Twin for Human-Robot Collaboration Tasks: A Case of Working Environment Configuration for the Robotic Lab,” 29th Annual Conf. for Computer-Aided Architectural Design Research in Asia (CAADRIA), pp. 381-390, 2024. https://doi.org/10.52842/conf.caadria.2024.3.381
16. [16] A. Egashira, Y. Horikawa, T. Hayashi, A. Kawamura, and R. Kurazume, “Near-future perception system: Previewed reality,” Adv. Robot., Vol.35, Issue 1, pp. 19-30, 2021. https://doi.org/10.1080/01691864.2020.1829041
17. [17] A. A. Malik and A. Brem, “Digital twins for collaborative robots: A case study in human-robot interaction,” Robot. Comput. Integr. Manuf., Vol.68, Article No.102092, 2021. https://doi.org/10.1016/j.rcim.2020.102092
18. [18] R. Etzi, S. Huang, G. W. Scurati, S. Lyu, F. Ferrise, A. Gallace, A. Gaggioli, A. Chirico, M. Carulli, and M. Bordegoni, “Using virtual reality to test human-robot interaction during a collaborative task,” Int. Design Engineering Technical Conf. and Computers and Infromation in Engineering Conf., 2019. https://doi.org/10.1115/DETC2019-97415
19. [19] V. Havard, B. Jeanne, M. Lacomblez, and D. Baudry, “Digital twin and virtual reality: A co-simulation environment for design and assessment of industrial workstations,” Production & Manufacturing Research, Vol.7, Issue 1, pp. 472-489, 2019. https://doi.org/10.1080/21693277.2019.1660283
20. [20] C. Peukert, J. Pfeiffer, M. Meißner, T. Pfeiffer, and C. Weinhardt, “Shopping in virtual reality stores: The influence of immersion on system adoption,” J. Manag. Inf. Syst., Vol.36, Issue 3, pp. 755-788, 2019. https://doi.org/10.1080/07421222.2019.1628889
21. [21] M. E. Walker, M. Gramopadhye, B. Ikeda, J. Burns, and D. Szafir, “The cyber-physical control room: A mixed reality interface for mobile robot teleoperation and human-robot teaming,” Proc. of the 2024 ACM/IEEE Int. Conf. on Human-Robot Interaction, pp. 762-771, 2024. https://doi.org/10.1145/3610977.3634981
22. [22] T. Inamura, “Digital twin of experience for human-robot collaboration through virtual reality,” Int. J. Automation Technol., Vol.17, No.3, pp. 284-291, 2023. https://doi.org/10.20965/ijat.2023.p0284
23. [23] T. Inamura and Y. Mizuchi, “SIGVerse: A cloud-based VR platform for research on multimodal human-robot interaction,” Frontiers in robotics and AI, Vol.8, Article No.549360, 2021. https://doi.org/10.3389/frobt.2021.549360
24. [24] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara, and K. Murase, “Development of human support robot as the research platform of a domestic mobile manipulator,” ROBOMECH J., Vol.6, No.1, Article No.4, 2019. urlhttps://doi.org/10.1186/s40648-019-0132-3
25. [25] T. van der Zant and L. Iocchi, “RoboCup@Home: Adaptive benchmarking of robot bodies and minds,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol.7072, pp. 214-225, 2011. https://doi.org/10.1007/978-3-642-25504-5_22
26. [26] W. Kritzinger, M. Karner, G. Traar, J. Henjes, and W. Sihn, “Digital twin in manufacturing: A categorical literature review and classification,” IFAC-PapersOnLine, Vol.51, Issue 11, pp. 1016-1022, 2018. https://doi.org/10.1016/j.ifacol.2018.08.474
27. [27] H. K. Kim, J. Park, Y. Choi, and M. Choe, “Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment,” Appl. Ergon., Vol.69, pp. 66-73, 2018. urlhttps://doi.org/10.1016/j.apergo.2017.12.016