Research Paper:
Semantic-Topological Cognitive Maps for Assistive Indoor Navigation of a Robot Partner
Anhar Risnumawan*,**
, Achmad Fahrul Aji*,***
, Raditya Artha Rochmanto*,***
, Teuku Zikri Fatahillah*
, and Naoyuki Kubota*,

*Department of Mechanical Systems Engineering, Graduate School of Systems Design, Tokyo Metropolitan University
6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan
Corresponding author
**Mechatronics Engineering Division, Politeknik Elektronika Negeri Surabaya
Jalan Raya ITS, Surabaya 60111, Indonesia
***Department of Electrical Engineering, Politeknik Negeri Semarang
Jl. Prof. H. Soedarto SH, Tembalang, Semarang 50275, Indonesia
Assistive indoor mobile robots are increasingly expected to connect natural-language requests about familiar places and objects with the metric maps used for localization and motion planning. Metric simultaneous localization and mapping (SLAM) supports reliable geometry and collision avoidance, whereas many open-vocabulary scene-graph systems remain separated from the navigation stacks used on deployed robots. This paper presents a semantic-topological cognitive map (STCM) that bridges these representations for assistive navigation. The system derives place nodes from a LiDAR–inertial pose stream through a dual growing neural gas backend and consolidates object instances over time. GroundingDINO and MobileSAM provide open-vocabulary semantic grounding in the map frame through LiDAR–image association and temporal score fusion. A tool-mediated large language model interface maps user commands to explicit STCM nodes, while graph search and ROS 2 Navigation2 (Nav2) remain responsible for path computation and execution. Evaluation with recorded robot sessions, an outdoor distribution-shift probe, and a public indoor-scene benchmark indicates that the STCM backend improves object-graph construction and that the language interface improves functional-command grounding when requested objects are recoverable by perception. The pipeline is demonstrated on a Ranger Garden Assistant platform with FAST-LIO2 mapping, incremental graph construction during the mapping pass, and waypoint execution from a serialized map artifact.
Ranger Garden Assistant platform with semantic topological cognitive map
- [1] P. A. Hancock, D. R. Billings, K. E. Schaefer, J. Y. C. Chen, E. J. de Visser, and R. Parasuraman, “A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction,” Human Factors, Vol.53, No.5, pp. 517-527, 2011. https://doi.org/10.1177/0018720811417254
- [2] A. A. Saputra, K. Wada, S. Masuda, and N. Kubota, “Multi-scopic Neuro-cognitive Adaptation for Legged Locomotion Robots,” Scientific Reports, Vol.12, Article No.16222, 2022. https://doi.org/10.1038/s41598-022-19599-2
- [3] E. C. Tolman, “Cognitive Maps in Rats and Men,” Psychological Review, Vol.55, No.4, pp. 189-208, 1948. https://doi.org/10.1037/h0061626
- [4] J. O’Keefe and L. Nadel, “The Hippocampus as a Cognitive Map,” Clarendon Press, 1978.
- [5] J. J. Gibson, “The Ecological Approach to Visual Perception,” Houghton Mifflin, 1979.
- [6] J. Woo and N. Kubota, “Recognition of Indoor Environment by Robot Partner Using Conversation,” J. Adv. Comput. Intell. Intell. Inform., Vol.17, No.5, pp. 753-760, 2013. https://doi.org/10.20965/jaciii.2013.p0753
- [7] N. Kubota and A. Yorita, “Topological Environment Reconstruction in Informationally Structured Space for Pocket Robot Partners,” Proc. IEEE Int. Symp. on Computational Intelligence in Robotics and Automation (CIRA), pp. 165-170, 2009. https://doi.org/10.1109/CIRA.2009.5423215
- [8] A. Rosinol, M. Abate, Y. Chang, and L. Carlone, “Kimera: An Open-Source Library for Real-Time Metric-Semantic Localization and Mapping,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), 2020. https://doi.org/10.1109/ICRA40945.2020.9196885
- [9] N. Hughes, Y. Chang, and L. Carlone, “Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization,” Proc. Robotics: Science and Systems (RSS), 2022. https://doi.org/10.15607/RSS.2022.XVIII.050
- [10] H. Bavle, J. L. Sánchez-López, M. Shaheer, J. Civera, and H. Voos, “S-Graphs+: Real-Time Localization and Mapping Leveraging Hierarchical Representations,” IEEE Robotics and Automation Letters, Vol.8, No.8, pp. 4927-4934, 2023. https://doi.org/10.1109/LRA.2023.3290512
- [11] C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual Language Maps for Robot Navigation,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 10608-10615, 2023. https://doi.org/10.1109/ICRA48891.2023.10160969
- [12] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li et al., “ConceptFusion: Open-Set Multimodal 3D Mapping,” Proc. Robotics: Science and Systems (RSS), 2023. https://doi.org/10.15607/RSS.2023.XIX.066
- [13] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal et al. “ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 5021-5028, 2024. https://doi.org/10.1109/ICRA57147.2024.10610243
- [14] F. L. Busch, T. Homberger, J. Ortega-Peimbert, Q. Yang, and O. Andersson, “One Map to Find Them All: Real-Time Open-Vocabulary Mapping for Zero-Shot Multi-Object Navigation,” arXiv preprint, arXiv:2409.11764, 2024. https://doi.org/10.48550/arXiv.2409.11764
- [15] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, “Open-Vocabulary Queryable Scene Representations for Real World Planning,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 11509-11522, 2023. https://doi.org/10.1109/ICRA48891.2023.10161534
- [16] D. Shah, B. Osiński, B. Ichter, and S. Levine, “LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action,” Proc. 6th Conf. on Robot Learning (CoRL), PMLR, Vol.205, pp. 492-504, 2022.
- [17] M. Chang, T. Gervet, M. Khanna, S. Yenamandra, D. Shah, S. Y. Min, K. Shah, C. Paxton, S. Gupta, D. Batra, R. Mottaghi, J. Malik, and D. S. Chaplot, “GOAT: GO to Any Thing,” Proc. Robotics: Science and Systems (RSS), 2024. https://doi.org/10.15607/RSS.2024.XX.073
- [18] Y. Tang, M. Wang, Y. Deng et al., “OpenIN: Open-Vocabulary Instance-Oriented Navigation in Dynamic Domestic Environments,” IEEE Robotics and Automation Letters, Vol.10, No.9, pp. 9256-9263, 2025. https://doi.org/10.1109/LRA.2025.3592071
- [19] S. Raychaudhuri, D. Ta, K. Ashton, A. X. Chang, J. Wang, and B. Bucher, “Zero-Shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation,” arXiv preprint, arXiv:2411.07848, 2024. https://doi.org/10.48550/arXiv.2411.07848
- [20] L. Zhang, X. Hao, Q. Xu et al., “MapNav: A Novel Memory Representation via Annotated Semantic Maps for VLM-based Vision-and-Language Navigation,” Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 13032-13056, 2025. https://doi.org/10.18653/v1/2025.acl-long.638
- [21] D. S. Chaplot et al., “Neural Topological SLAM for Visual Navigation,” Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. https://doi.org/10.1109/CVPR42600.2020.01289
- [22] D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. He, and L. Wang, “ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments,” IEEE Trans. Pattern Anal. Mach. Intell., Vol.47, No.7, pp. 5130-5145, 2025. https://doi.org/10.1109/TPAMI.2024.3386695
- [23] S. Garg, K. Rana, M. Hosseinzadeh, L. Mares, N. Sünderhauf, F. Dayoub, and I. Reid, “RoboHop: Segment-Based Topological Map Representation for Open-World Visual Navigation,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 4090-4097, 2024. https://doi.org/10.1109/ICRA57147.2024.10610234
- [24] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “FAST-LIO2: Fast Direct LiDAR-Inertial Odometry,” IEEE Trans. Robotics, Vol.38, No.4, pp. 2053-2073, 2022. https://doi.org/10.1109/TRO.2022.3141876
- [25] B. Fritzke, “A Growing Neural Gas Network Learns Topologies,” Advances in Neural Information Processing Systems (NeurIPS), Vol.7, pp. 625-632, 1994.
- [26] F. Ardilla, A. A. Saputra, and N. Kubota, “Multi-Scale Batch-Learning Growing Neural Gas Efficiently for Dynamic Data Distributions,” Int. J. Automation Technol., Vol.17, No.3, pp. 206-216, 2023. https://doi.org/10.20965/ijat.2023.p0206
- [27] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang, “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection,” Proc. European Conf. on Computer Vision (ECCV), pp. 38-55, 2024. https://doi.org/10.1007/978-3-031-72970-6_3
- [28] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment Anything,” Proc. IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 3992-4003, 2023. https://doi.org/10.1109/ICCV51070.2023.00371
- [29] C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong, “Faster Segment Anything: Towards Lightweight SAM for Mobile Applications,” arXiv preprint, arXiv:2306.14289, 2023. https://doi.org/10.48550/arXiv.2306.14289
- [30] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,” Proc. 6th Conf. on Robot Learning (CoRL), PMLR, Vol.205, pp. 287-318, 2022.
- [31] H. Zhang, H. Deng, J. Ou, and C. Feng, “Mitigating Spatial Hallucination in Large Language Models for Path Planning via Prompt Engineering,” Sci. Rep., Vol.15, Article No.8881, 2025. https://doi.org/10.1038/s41598-025-93601-5
- [32] S. Macenski et al., “The Marathon 2: A Navigation System,” Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2020. https://doi.org/10.1109/IROS45743.2020.9341207
- [33] I. Armeni, Z.-Y. He, A. Zamir, J. Gwak, J. Malik, M. Fischer, and S. Savarese, “3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera,” Proc. IEEE Int. Conf. on Computer Vision (ICCV), pp. 5664-5673, 2019. https://doi.org/10.1109/ICCV.2019.00576
- [34] U.-H. Kim, J.-M. Park, T.-J. Song, and J.-H. Kim, “3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents,” IEEE Trans. Cybern., Vol.50, No.12, pp. 4921-4933, 2020. https://doi.org/10.1109/TCYB.2019.2931042
- [35] Y. Tian, Y. Chang, F. H. Arias, C. Nieto-Granda, J. P. How, and L. Carlone, “Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems,” IEEE Trans. on Robotics, Vol.38, No.4, pp. 2022-2038, 2022. https://doi.org/10.1109/TRO.2021.3137751
- [36] S. H. Allu, I. Kadosh, T. Summers, and Y. Xiang, “A Modular Robotic System for Autonomous Exploration and Semantic Updating in Large-Scale Indoor Environments,” arXiv preprint, arXiv:2409.15493, 2024. https://doi.org/10.48550/arXiv.2409.15493
- [37] N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 42-48, 2024. https://doi.org/10.1109/ICRA57147.2024.10610712
- [38] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov, “Object Goal Navigation Using Goal-Oriented Semantic Exploration,” Advances in Neural Information Processing Systems (NeurIPS), 2020.
- [39] N. Zantout, H. Zhang, P. Kachana, J. Qiu, G. Chen, J. Zhang, and W. Wang, “SORT3D: Spatial Object-Centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models,” Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 2201-2208, 2025. https://doi.org/10.1109/IROS60139.2025.11246145
- [40] X. Huang, S. Zhao, Y. Wang et al., “MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation,” arXiv preprint, arXiv:2511.10376, 2025. https://doi.org/10.48550/arXiv.2511.10376
- [41] H. Zhou, X. Wang, H. Li et al., “LagMemo: Language 3D Gaussian Splatting Memory for Multi-modal Open-Vocabulary Multi-Goal Visual Navigation,” arXiv preprint, arXiv:2510.24118, 2025. https://doi.org/10.48550/arXiv.2510.24118
- [42] J. Crespo, J. C. Castillo, O. M. Mozos et al., “Semantic Information for Robot Navigation: A Survey,” Applied Sciences, Vol.10, No.2, Article No.497, 2020. https://doi.org/10.3390/app10020497
- [43] R. Alqobali, M. Alshmrani, R. Alnasser et al., “A Survey on Robot Semantic Navigation Systems for Indoor Environments,” Applied Sciences, Vol.14, No.1, Article No.89, 2024. https://doi.org/10.3390/app14010089
- [44] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10371-10381, 2024. https://doi.org/10.1109/CVPR52733.2024.00987
- [45] A. Junaedy, H. Masuta, Y. Fuse, K. Sawai, T. Motoyoshi, and N. Takagi, “Online Topological Mapping on a Quadcopter with Fast Growing Neural Gas,” J. Adv. Comput. Intell. Intell. Inform., Vol.28, No.6, pp. 1354-1366, 2024. https://doi.org/10.20965/jaciii.2024.p1354
- [46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models from Natural Language Supervision,” Proc. Int. Conf. on Machine Learning (ICML), PMLR, Vol.139, pp. 8748-8763, 2021.
- [47] J. Long, E. Shelhamer, and T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015. https://doi.org/10.1109/cvpr.2015.7298965
- [48] D. Eigen and R. Fergus, “Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture,” Proc. IEEE Int. Conf. on Computer Vision (ICCV), pp. 2650-2658, 2015. https://doi.org/10.1109/iccv.2015.304
- [49] C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, “FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture,” Asian Conf. on Computer Vision (ACCV), pp. 213-228, 2016. https://doi.org/10.1007/978-3-319-54181-5_14
- [50] S. Lee, S.-J. Park, and K.-S. Hong, “RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation,” Proc. IEEE Int. Conf. on Computer Vision (ICCV), pp. 4990-4999, 2017. https://doi.org/10.1109/iccv.2017.533
- [51] J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 4628-4635, 2017. https://doi.org/10.1109/icra.2017.7989538
- [52] X. Hu, K. Yang, L. Fei, and K. Wang, “ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation,” Proc. IEEE Int. Conf. on Image Processing (ICIP), pp. 1440-1444, 2019. https://doi.org/10.1109/icip.2019.8803025
- [53] X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, “Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation,” European Conf. on Computer Vision (ECCV), pp. 561-577, 2020. https://doi.org/10.1007/978-3-030-58621-8_33
- [54] D. Seichter, M. Köhler, B. Lewandowski, T. Wengefeld, and H.-M. Gross, “Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis,” Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 13525-13531, 2021. https://doi.org/10.1109/ICRA48506.2021.9561675
- [55] K. Muravyev and K. Yakovlev, “Evaluation of Topological Mapping Methods in Indoor Environments,” IEEE Access, Vol.11, pp. 132683-132698, 2023. https://doi.org/10.1109/ACCESS.2023.3335818
- [56] B. Kuipers, “The Spatial Semantic Hierarchy,” Artif. Intell., Vol.119, Nos.1-2, pp. 191-233, 2000. https://doi.org/10.1016/S0004-3702(00)00017-5
- [57] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation,” Proc. AAAI Conf. on Artificial Intelligence, pp. 1507-1514, 2011. https://doi.org/10.1609/aaai.v25i1.7979
- [58] S. Stange and S. Kopp, “Effects of a Social Robot’s Self-Explanations on How Humans Understand and Evaluate Its Behavior,” Proc. ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI), pp. 619-627, 2020. https://doi.org/10.1145/3319502.3374802
- [59] O. Sotomi, D. Kodi, and A. Arab, “Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models,” arXiv preprint, arXiv:2504.05477, 2025. https://doi.org/10.48550/arXiv.2504.05477
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.