Service robots, in contrast to industrial robots, are devices aimed to operate in the service sector in replacement of or collaboration with human performers—in particular, domestic service robots carry out daily household chores. Recently, their popularity has increased and the range and difficulty of the activities they can perform are reaching amazing performances. There have been proposed several systems that can generalize single tasks such as object manipulation or scene understanding but they still fail on complex task planning, i.e., when and where they can perform such tasks. In this work, we propose the use of expert systems as a core module to understand user commands, infer the task context, request missing information, and plan action where each step of the plan may consist of basic state machines to generalized deep-learning models.

1. [1] L. Contreras and H. Okada, “Personal and Domestic Robotics,” Encyclopedia of Robotics, 2022.
2. [2] T. Inamura and Y. Mizuchi, “SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction,” Front. Robot. AI, Vol.8, Article No.549360, 2021. https://doi.org/10.3389/frobt.2021.549360
3. [3] L. Contreras, T. Yamamoto, Y. Matsusaka, and H. Okada, “Towards general purpose service robots: World Robot Summit – Partner Robot Challenge,” Advanced Robotics, Vol.36, Nos.17-18, pp. 812-824, 2022. https://doi.org/10.1080/01691864.2022.2109428
4. [4] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “RLBench: The Robot Learning Benchmark and Learning Environment,” IEEE Robotics and Automation Letters, Vol.5, Issue 2, pp. 3019-3026, 2020. https://doi.org/10.1109/LRA.2020.2974707
5. [5] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee, “Transporter Networks: Rearranging the Visual World for Robotic Manipulation,” Conf. on Robot Learning (CoRL), 2020.
6. [6] V. Kumar, R. Shah, G. Zhou, V. Moens, V. Caggiano, J. Vakil, A. Gupta, and A. Rajeswaran, “RoboHive: A Unified Framework for Robot Learning,” NeurIPS: Conf. on Neural Information Processing Systems, 2023.
7. [7] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra, “Habitat 2.0: Training Home Assistants to Rearrange their Habitat,” Proc. of the 35th Int. Conf. on Neural Information Processing Systems (NIPS’21), Article No.20, pp. 251-266, 2021.
8. [8] X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y. Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y. Min, V. Vondrus, T. Gervet, V.-P. Berges, J. M. Turner, O. Maksymets, Z. Kira, M. Kalakrishnan, J. Malik, D. S. Chaplot, U. Jain, D. Batra, A. Rai, and R. Mottaghi, “Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots,” arXiv preprint, arXiv:2310.13724, 2023. https://doi.org/10.48550/arXiv.2310.13724
9. [9] L. Contreras, Y. Matsusaka, T. Yamamoto, and H. Okada, “sDSPL – Towards a benchmark for general-purpose task evaluation in domestic service robots,” The 39th Annual Conf. of the Robotics Society of Japan, 2021.
10. [10] S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei, “BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments,” Conf. on Robot Learning (CoRL), 2021.
11. [11] C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, C. K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese, “iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks,” Conf. on Robot Learning (CoRL), 2021.
12. [12] M. Shridhar, L. Manuelli, and D. Fox, “CLIPort: What and Where Pathways for Robotic Manipulation,” Proc. of the 5th Conf. on Robot Learning (CoRL), 2021.
13. [13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” Proc. of the 38th Int. Conf. on Machine Learning, 2021.
14. [14] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” arXiv preprint, arXiv:2307.15818, 2023. https://doi.org/10.48550/arXiv.2307.15818
15. [15] I. Leal, K. Choromanski, D. Jain, A. Dubey, J. Varley, M. Ryoo, Y. Lu, F. Liu, V. Sindhwani, Q. Vuong, T. Sarlos, K. Oslund, K. Hausman, and K. Rao, “SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention,” 2024 IEEE Int. Conf. on Robotics and Automation (ICRA), 2024. https://doi.org/10.1109/ICRA57147.2024.10611597
16. [16] C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual Language Maps for Robot Navigation,” 2023 IEEE Int. Conf. on Robotics and Automation (ICRA), 2023. https://doi.org/10.1109/ICRA48891.2023.10160969
17. [17] Y. Zhou, L. Huang, Q. Bu, J. Zeng, T. Li, H. Qiu, H. Zhu, M. Guo, Y. Qiao, and H. Li, “Embodied Understanding of Driving Scenarios,” arXiv preprint, arXiv:2403.04593, 2024. https://doi.org/10.48550/arXiv.2403.04593
18. [18] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas, “A Generalist Agent,” arXiv preprint, arXiv:2205.06175, 2022. https://doi.org/10.48550/arXiv.2205.06175
19. [19] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, Md. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep Learning Scaling is Predictable, Empirically,” arXiv preprint, arXiv:1712.00409, 2017. https://doi.org/10.48550/arXiv.1712.00409
20. [20] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv preprint, arXiv:2001.08361, 2020. https://doi.org/10.48550/arXiv.2001.08361
21. [21] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “TidyBot: Personalized Robot Assistance with Large Language Models,” IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2023. https://doi.org/10.1109/IROS55552.2023.10341577
22. [22] P. Liu, Y. Orru, J. Vakil, C. Paxton, N. M. M. Shafiullah, and L. Pinto, “OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics,” arXiv preprint, arXiv:2401.12202, 2024. https://doi.org/10.48550/arXiv.2401.12202
23. [23] Z. Hu, F. Lucchetti, C. Schlesinger, Y. Saxena, A. Freeman, S. Modak, A. Guha, and J. Biswas, “Deploying and Evaluating LLMs to Program Service Mobile Robots,” IEEE Robotics and Automation Letters, Vol.9, Issue 3, pp. 2853-2860, 2024. https://doi.org/10.1109/LRA.2024.3360020
24. [24] K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, and S. Kambhampati, “PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change,” arXiv preprint, arXiv:2206.10498, 2023. https://doi.org/10.48550/arXiv.2206.10498
25. [25] K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati, “Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change),” 2022.
26. [26] Y. Zhang, H. Bai, R. Zhang, J. Gu, S. Zhai, J. Susskind, and N. Jaitly, “How Far Are We from Intelligent Visual Deductive Reasoning?,” arXiv preprint, arXiv:2403.04732, 2024. https://doi.org/10.48550/arXiv.2403.04732
27. [27] A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. D.-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu, “Teaching Large Language Models to Reason with Reinforcement Learning,” arXiv preprint, arXiv:2403.04642, 2024. https://doi.org/10.48550/arXiv.2403.04642
28. [28] R. C. Schank, “Conceptual dependency: A theory of natural language understanding,” Cognitive Psychology, Vol.3, Issue 4, pp. 552-631, 1972. https://doi.org/10.1016/0010-0285(72)90022-9
29. [29] J. Savage, D. Rosenblueth, M. Matamoros, M. Negrete, L. Contreras, J. Cruz, R. Martell, H. Estrada, and H. Okada, “Semantic reasoning in service robots using expert systems,” Robotics and Autonomous Systems, Vol.114, pp. 77-92, 2019. https://doi.org/10.1016/j.robot.2019.01.007
30. [30] E. Vazquez, L. Contreras, H. Okada, and N. Iwahashi, “Multimodal human intention-driven robot motion control in collaborative tasks,” The 38th Annual Conf. of the Robotics Society of Japan, 2020.
31. [31] L. Contreras, Y. Muto, H. Okada, and A. Sakamaki, “Robot-object interaction strategies for object recognition,” The 37th Annual Conf. of the Robotics Society of Japan, 2019.
32. [32] L. Contreras, H. Yokoyama, and H. Okada, “Multimodal feedback for active perception in service robots,” IEEE/RSJ IROS Workshops, 2018.
33. [33] G. Riley, “CLIPS Reference Manual Version 6.0. Technical Report Number JSC-25012,” Software Technology Branch, Lyndon B. Johnson Space Center, 1994.
34. [34] G. Riley, “Adventures in Rule-Based Programming: A CLIPS Tutorial,” Secret Society Software, LLC, 2022.
35. [35] S. Ortuno-Chanelo, L. Contreras, J. Savage, and H. Okada, “Keep it Simple: Understanding Natural Language Commands for General-Purpose Service Robots,” IEEE/SICE Int. Symp. on System Integration, 2024. https://doi.org/10.1109/SII58957.2024.10417341
36. [36] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara, and K. Murase, “Development of human support robot as the research platform of a domestic mobile manipulator,” ROBOMECH J., Vol.6, Article No.4, 2019. https://doi.org/10.1186/s40648-019-0132-3
37. [37] L. Contreras, J. Savage, S. Ortuno-Chanelo, M. Negrete, A. Sakamaki, and H. Okada, “Fail It till You Make It: Error Expectation in Complex-Plan Execution for Service Robots,” Int. Conf. on Interactive Collaborative Robotics, pp. 36-46, 2023. https://doi.org/10.1007/978-3-031-43111-1_4