Robots capable of accurately understanding situations based on environmental information and collaborating effectively with humans in a variety of tasks are in high demand. Deep-learning-based approaches can be effective for realizing such collaborative robots; however, collecting data and achieving high generalization performance are challenges. To address this issue, we utilize “play data” collected by teleoperating a robot when performing collaborative tasks with a human partner. This study proposes a play-supervised latent goal for human–robot collaboration (play-HRC) framework that learns from unlabeled play data for human–robot collaborative scenarios without explicit goal observation. Unlike play-supervised latent motor plan that generates actions conditioned on an externally specified goal observation, play-HRC must infer a latent goal from an initial state. At the beginning of the task execution, play-HRC infers an achievable goal from its initial state. During execution, it generates actions and predicts future environmental states while adaptively optimizing the latent goal by minimizing prediction errors. To evaluate the performance of play-HRC, we conducted tests in a real-world environment using a task in which a robot and human collaboratively manipulate objects. The results highlighted the ability of the framework to infer achievable goals, generate actions, and dynamically adjust goals in response to human behavior.

1. [1] D. Mukherjee, K. Gupta, L. H. Chang, and H. Najjaran, “A survey of robot learning strategies for human-robot collaboration in industrial settings,” Robot. Comput.-Integr. Manuf., Vol.73, Article No.102231, 2022. https://doi.org/10.1016/j.rcim.2021.102231
2. [2] A. Bauer, D. Wollherr, and M. Buss, “Human–robot collaboration: A survey,” Int. J. Humanoid Robot., Vol.5, No.1, pp. 47-66, 2008. https://doi.org/10.1142/S0219843608001303
3. [3] L. Contreras, J. Savage, S. Ortuño-Chanelo, M. Negrete, and H. Okada, “Complex task planning for general-purpose service robots,” J. Robot. Mechatron., Vol.37, No.3, pp. 594-603, 2025. https://doi.org/10.20965/jrm.2025.p0594
4. [4] C. Wang et al., “MimicPlay: Long-horizon imitation learning by watching human play,” Proc. 7th Conf. Robot Learn., pp. 201-221, 2023.
5. [5] A. Brohan et al., “RT-1: Robotics transformer for real-world control at scale,” Robotics: Science and Systems XIX, 2023. https://doi.org/10.15607/RSS.2023.XIX.025
6. [6] B. Zitkovich et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” Proc. 7th Conf. Robot Learn., pp. 2165-2183, 2023.
7. [7] T. Motoda, T. Kitamura, R. Hanai, and Y. Domae, “SuctionPrompt: Visual-assisted robotic picking with a suction cup using vision-language models and facile hardware design,” J. Robot. Mechatron., Vol.37, No.2, pp. 374-386, 2025. https://doi.org/10.20965/jrm.2025.p0374
8. [8] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” Int. J. Robot. Res., Vol.32, No.11, pp. 1238-1274, 2013. https://doi.org/10.1177/0278364913495721
9. [9] J. Ibarz et al., “How to train your robot with deep reinforcement learning: Lessons we have learned,” Int. J. Robot. Res., Vol.40, Nos.4-5, pp. 698-721, 2021. https://doi.org/10.1177/0278364920987859
10. [10] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, “Learning and generalization of motor skills by learning from demonstration,” 2009 IEEE Int. Conf. Robot. Autom., pp. 763-768, 2009. https://doi.org/10.1109/ROBOT.2009.5152385
11. [11] T. Zhang et al., “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” 2018 IEEE Int. Conf. Robot. Autom., pp. 5628-5635, 2018. https://doi.org/10.1109/ICRA.2018.8461249
12. [12] T. Yamashita, H. Suzuki, and R. Tasaki, “Skill implemented motion planning for precision assembly task of part mating gears,” J. Robot. Mechatron., Vol.37, No.2, pp. 434-443, 2025. https://doi.org/10.20965/jrm.2025.p0434
13. [13] C. Lynch et al., “Learning latent plans from play,” Proc. 3rd Annu. Conf. Robot Learn., pp. 1113-1132, 2019.
14. [14] R. P. N. Rao and D. H. Ballard, “Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects,” Nat. Neurosci., Vol.2, No.1, pp. 79-87, 1999. https://doi.org/10.1038/4580
15. [15] A. Clark, “Whatever next? Predictive brains, situated agents, and the future of cognitive science,” Behav. Brain Sci., Vol.36, No.3, pp. 181-204, 2013. https://doi.org/10.1017/S0140525X12000477
16. [16] A. Ciria, G. Schillaci, G. Pezzulo, V. V. Hafner, and B. Lara, “Predictive processing in cognitive robotics: A review,” Neural Comput., Vol.33, No.5, pp. 1402-1432, 2021. https://doi.org/10.1162/neco_a_01383
17. [17] T. Taniguchi et al., “World models and predictive coding for cognitive and developmental robotics: Frontiers and challenges,” Adv. Robot., Vol.37, No.13, pp. 780-806, 2023. https://doi.org/10.1080/01691864.2023.2225232
18. [18] J. Tani, “Learning to generate articulated behavior through the bottom-up and the top-down interaction processes,” Neural Netw., Vol.16, No.1, pp. 11-23, 2003. https://doi.org/10.1016/S0893-6080(02)00214-9
19. [19] S. Murata et al., “Learning to perceive the world as probabilistic or deterministic via interaction with others: A neuro-robotics experiment,” IEEE Trans. Neural Netw. Learn. Syst., Vol.28, No.4, pp. 830-848, 2017. https://doi.org/10.1109/TNNLS.2015.2492140
20. [20] P. Englert and M. Toussaint, “Learning manipulation skills from a single demonstration,” Int. J. Robot. Res., Vol.37, No.1, pp. 137-154, 2018. https://doi.org/10.1177/0278364917743795
21. [21] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” Proc. 1st Conf. Robot Learn., pp. 357-368, 2017.
22. [22] N. M. M. Shafiullah et al., “On bringing robots home,” 2023. https://doi.org/10.48550/arXiv.2311.16098
23. [23] D. Ghosh et al., “Octo: An open-source generalist robot policy,” Robotics: Science and Systems XX, 2024. https://doi.org/10.15607/rss.2024.xx.090
24. [24] A. O’Neill et al., “Open X-Embodiment: Robotic learning datasets and RT-X models: Open X-Embodiment Collaboration,” 2024 IEEE Int. Conf. Robot. Autom., pp. 6892-6903, 2024. https://doi.org/10.1109/ICRA57147.2024.10611477
25. [25] C. Xu et al., “RL Token: Bootstrapping online RL with vision-language-action models,” arXiv:2604.23073, 2026. https://doi.org/10.48550/arXiv.2604.23073
26. [26] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard, “Latent plans for task-agnostic offline reinforcement learning,” Proc. 6th Conf. Robot Learn., pp. 1838-1849, 2023.
27. [27] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto, “From play to policy: Conditional behavior generation from uncurated robot data,” arXiv:2210.10047, 2022. https://doi.org/10.48550/arXiv.2210.10047
28. [28] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3M: A universal visual representation for robot manipulation,” arXiv:2203.12601, 2022. https://doi.org/10.48550/arXiv.2203.12601
29. [29] S. Li, L. Zhang, and X. Diao, “Deep-learning-based human intention prediction using RGB images and optical flow,” J. Intell. Robot. Syst., Vol.97, No.1, pp. 95-107, 2020. https://doi.org/10.1007/s10846-019-01049-3
30. [30] L. Geng, W. Yang, Y. Jiao, S. Zeng, and X. Chen, “A multilayer human motion prediction perceptron by aggregating repetitive motion,” Mach. Vis. Appl., Vol.34, No.6, Article No.98, 2023. https://doi.org/10.1007/s00138-023-01447-6
31. [31] J. Tang, J. Wang, and J.-F. Hu, “Predicting human poses via recurrent attention network,” Vis. Intell., Vol.1, No.1, Article No.18, 2023. https://doi.org/10.1007/s44267-023-00020-z
32. [32] P. Schydlo, M. Rakovic, L. Jamone, and J. Santos-Victor, “Anticipation in human-robot cooperation: A recurrent neural network approach for multiple action sequences prediction,” 2018 IEEE Int. Conf. Robot. Autom., pp. 5909-5914, 2018. https://doi.org/10.1109/ICRA.2018.8460924
33. [33] S. Murata et al., “Achieving human–robot collaboration with dynamic goal inference by gradient descent,” Proc. 26th Int. Conf. Neural Inf. Process., Part 2, pp. 579-590, 2019. https://doi.org/10.1007/978-3-030-36711-4_49
34. [34] A. Ahmadi and J. Tani, “A novel predictive-coding-inspired variational RNN model for online prediction and recognition,” Neural Comput., Vol.31, No.11, pp. 2025-2074, 2019. https://doi.org/10.1162/neco_a_01228
35. [35] K. Friston, “The free-energy principle: A unified brain theory?,” Nat. Rev. Neurosci., Vol.11, No.2, pp. 127-138, 2010. https://doi.org/10.1038/nrn2787
36. [36] M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks.” Proc. 36th Int. Conf. Mach. Learn., pp. 6105-6114, 2019.
37. [37] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” Proc. 2nd Int. Conf. Learn. Represent., 2014. https://doi.org/10.48550/arXiv.1312.6114
38. [38] A. Vaswani et al., “Attention is all you need,” Proc. 31st Int. Conf. Neural Inf. Process. Syst., pp. 6000-6010, 2017.
39. [39] K. Cho et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” Proc. 2014 Conf. Empir. Methods Nat. Lang. Process., pp. 1724-1734, 2014. https://doi.org/10.3115/v1/D14-1179
40. [40] I. Higgins et al., “beta-VAE: Learning basic visual concepts with a constrained variational framework,” 5th Int. Conf. Learn. Represent., pp. 60-81, 2017.
41. [41] O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning over unstructured data,” IEEE Robot. Autom. Lett., Vol.7, No.4, pp. 11205-11212, 2022. https://doi.org/10.1109/LRA.2022.3196123
42. [42] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering Atari with discrete world models,” arXiv:2010.02193. 2020. https://doi.org/10.48550/arXiv.2010.02193
43. [43] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 7th Int. Conf. Learn. Represent., 2019.
44. [44] D. Zhang, D. Dai, Y. He, F. S. Bao, and B. Xie, “RLScheduler: An automated HPC batch job scheduler using reinforcement learning,” Int. Conf. High Perform. Comput. Netw. Storage Anal., 2020. https://doi.org/10.1109/SC41405.2020.00035