FFD-SLAM: A Real-Time Visual SLAM Toward Dynamic Scenes with Semantic and Optical Flow Information

Hao Zhang; Yu Wang; Tianjie Zhong; Fangyan Dong; Kewei Chen

doi:10.20965/jaciii.2024.p0586

single-jc.php

« previous

JACIII Vol.28 No.3 pp. 586-594

doi: 10.20965/jaciii.2024.p0586

(2024)

Research Paper:

Views over last 60 days: 227

FFD-SLAM: A Real-Time Visual SLAM Toward Dynamic Scenes with Semantic and Optical Flow Information

Hao Zhang^*, Yu Wang^**, Tianjie Zhong^, Fangyan Dong^, and Kewei Chen^*,†

^*Faculty of Mechanical Engineering & Mechanics, Ningbo University
No.818 Fenghua Road, Jiangbei District, Ningbo, Zhejiang 315211, China

^†Corresponding author

^**China Academy of Safety Science & Technology
Building A1, No.32 Beiyuan Road, Chaoyang District, Beijing 100012, China

Received:

December 18, 2023

Accepted:

January 18, 2024

Published:

May 20, 2024

Keywords:

dynamic object detection, optical flow filtering, image feature points elimination, SLAM algorithm

Abstract

To solve the problem of poor localization accuracy and robustness of visual simultaneous localization and mapping (SLAM) systems in highly dynamic environments, this paper proposes a dynamic visual SLAM algorithm called FFD-SLAM that fuses the target detection network with the optical flow method. The algorithm considers ORB-SLAM2 as the basic framework, joins the semantic thread in parallel with its tracking thread, initially obtains the set of feature points through the real-time detection of dynamic objects in the environment through YOLOv5 in the semantic thread, then filters the set of feature points obtained in the semantic thread through the optical flow module, and finally utilizes the remaining static feature points for the matching calculation. Experiments showed that the proposed algorithm showed an improvement of approximately 97% in the localization accuracy compared with the ORB-SLAM2 algorithm in a highly dynamic environment, which effectively improves the localization accuracy and robustness of the system. The proposed algorithm also showed a higher real-time performance compared with some excellent dynamic SLAM algorithms.

Cite this article as:

H. Zhang, Y. Wang, T. Zhong, F. Dong, and K. Chen, “FFD-SLAM: A Real-Time Visual SLAM Toward Dynamic Scenes with Semantic and Optical Flow Information,” J. Adv. Comput. Intell. Intell. Inform., Vol.28 No.3, pp. 586-594, 2024.

Data files:

References

[1] S. Yuan, H. Wang, and L. Xie, “Survey on Localization Systems and Algorithms for Unmanned Systems,” Unmanned Systems, Vol.9, No.2, pp. 129-163, 2021. https://doi.org/10.1142/S230138502150014X
[2] H. Wang, C. Wang, and L. Xie, “Intensity-SLAM: Intensity Assisted Localization and Mapping for Large Scale Environment,” IEEE Robotics and Automation Letters, Vol.6, No.2, pp. 1715-1721, 2021. https://doi.org/10.1109/LRA.2021.3059567
[3] Y. Fan, Q. Zhang, S. Liu, Y. Tang, X. Jing, J. Yao, and H. Han, “Semantic SLAM with More Accurate Point Cloud Map in Dynamic Environments,” IEEE Access, Vol.8, pp. 112237-112252, 2020. https://doi.org/10.1109/ACCESS.2020.3003160
[4] K. Wang, X. Yao, Y. Huang, M. Liu, and Y. Lu, “Review of Visual SLAM in Dynamic Environment,” Robot, Vol.43, No.6, Article No.715732, 2021. https://doi.org/10.13973/j.cnki.robot.200468
[5] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, “Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age,” IEEE Trans. on Robotics, Vol.32, No.6, pp. 1309-1332, 2016. https://doi.org/10.1109/TRO.2016.2624754
[6] R. Mur-Artal and J. D. Tardos, “ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras,” IEEE Trans. on Robotics, Vol.33, No.5, pp. 1255-1262, 2017. https://doi.org/10.1109/TRO.2017.2705103
[7] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Seg-mentation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.39, No.12, pp. 24810-2495, 2017. https://doi.org/10.1109/TPAMI.2016.2644615
[8] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” 2017 IEEE Int. Conf. on Computer Vision (ICCV), pp. 2961-2969, 2017. https://doi.org/10.48550/arXiv.1703.06870
[9] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” Proc. of 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. https://doi.org/10.1109/CVPR.2016.91
[10] W. Liu, D. Anguelov, D. Erhan et al., “SSD: Single Shot MultiBox Detector,” European Conf. on Computer Vision, pp. 21-37, 2016. https://doi.org/10.1007/978-3-319-46448-0_2
[11] Y. Liu and J. Miura, “RDS-SLAM: Real-Time Dynamic SLAM Using Semantic Segmentation Methods,” IEEE Access, Vol.9, pp. 23772-23785, 2021. https://doi.org/10.1109/ACCESS.2021.3050617
[12] C. Yu, Z. Liu, X. Liu et al., “DS-SLAM: A Semantic Visual SLAM Towards Dynamic Environments,” Proc. of 2018 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 1168-1174, 2018. https://doi.org/10.1109/IROS.2018.8593691
[13] B. Bescos, J. M. Facil, J. Civera, and J. Neira, “DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes,” IEEE Robotics and Automation Letters, Vol.3, No.4, pp. 4076-4083, 2018. https://doi.org/10.1109/LRA.2018.2860039
[14] P. Su, S. Y. Luo, and X. C. Huang, “Real-Time Dynamic SLAM Algorithm Based on Deep Learning,” IEEE Access, Vol.10, pp. 87754-87766, 2022. https://doi.org/10.1109/ACCESS.2022.3199350
[15] F. W. Zhong, S. Wang, Z. Q. Zhang et al., “Detect-SLAM: Making Object Detection and SLAM Mutually Beneficial,” IEEE Winter Conf. on Applications of Computer Vision, pp. 1001-1010, 2018. https://doi.org/10.1109/WACV.2018.00115
[16] B. D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. of 7th Int. Joint Conf. on Artificial Intelligence, pp. 456-459, 1981.
[17] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” Proc. of 2012 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 573-580, 2012. https://doi.org/10.1109/IROS.2012.6385773

This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

[1] [1] S. Yuan, H. Wang, and L. Xie, “Survey on Localization Systems and Algorithms for Unmanned Systems,” Unmanned Systems, Vol.9, No.2, pp. 129-163, 2021. https://doi.org/10.1142/S230138502150014X

[2] [2] H. Wang, C. Wang, and L. Xie, “Intensity-SLAM: Intensity Assisted Localization and Mapping for Large Scale Environment,” IEEE Robotics and Automation Letters, Vol.6, No.2, pp. 1715-1721, 2021. https://doi.org/10.1109/LRA.2021.3059567

[3] [3] Y. Fan, Q. Zhang, S. Liu, Y. Tang, X. Jing, J. Yao, and H. Han, “Semantic SLAM with More Accurate Point Cloud Map in Dynamic Environments,” IEEE Access, Vol.8, pp. 112237-112252, 2020. https://doi.org/10.1109/ACCESS.2020.3003160

[4] [4] K. Wang, X. Yao, Y. Huang, M. Liu, and Y. Lu, “Review of Visual SLAM in Dynamic Environment,” Robot, Vol.43, No.6, Article No.715732, 2021. https://doi.org/10.13973/j.cnki.robot.200468

[5] [5] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, “Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age,” IEEE Trans. on Robotics, Vol.32, No.6, pp. 1309-1332, 2016. https://doi.org/10.1109/TRO.2016.2624754

[6] [6] R. Mur-Artal and J. D. Tardos, “ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras,” IEEE Trans. on Robotics, Vol.33, No.5, pp. 1255-1262, 2017. https://doi.org/10.1109/TRO.2017.2705103

[7] [7] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Seg-mentation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.39, No.12, pp. 24810-2495, 2017. https://doi.org/10.1109/TPAMI.2016.2644615

[8] [8] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” 2017 IEEE Int. Conf. on Computer Vision (ICCV), pp. 2961-2969, 2017. https://doi.org/10.48550/arXiv.1703.06870

[9] [9] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” Proc. of 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. https://doi.org/10.1109/CVPR.2016.91

[10] [10] W. Liu, D. Anguelov, D. Erhan et al., “SSD: Single Shot MultiBox Detector,” European Conf. on Computer Vision, pp. 21-37, 2016. https://doi.org/10.1007/978-3-319-46448-0_2

[11] [11] Y. Liu and J. Miura, “RDS-SLAM: Real-Time Dynamic SLAM Using Semantic Segmentation Methods,” IEEE Access, Vol.9, pp. 23772-23785, 2021. https://doi.org/10.1109/ACCESS.2021.3050617

[12] [12] C. Yu, Z. Liu, X. Liu et al., “DS-SLAM: A Semantic Visual SLAM Towards Dynamic Environments,” Proc. of 2018 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 1168-1174, 2018. https://doi.org/10.1109/IROS.2018.8593691

[13] [13] B. Bescos, J. M. Facil, J. Civera, and J. Neira, “DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes,” IEEE Robotics and Automation Letters, Vol.3, No.4, pp. 4076-4083, 2018. https://doi.org/10.1109/LRA.2018.2860039

[14] [14] P. Su, S. Y. Luo, and X. C. Huang, “Real-Time Dynamic SLAM Algorithm Based on Deep Learning,” IEEE Access, Vol.10, pp. 87754-87766, 2022. https://doi.org/10.1109/ACCESS.2022.3199350

[15] [15] F. W. Zhong, S. Wang, Z. Q. Zhang et al., “Detect-SLAM: Making Object Detection and SLAM Mutually Beneficial,” IEEE Winter Conf. on Applications of Computer Vision, pp. 1001-1010, 2018. https://doi.org/10.1109/WACV.2018.00115

[16] [16] B. D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. of 7th Int. Joint Conf. on Artificial Intelligence, pp. 456-459, 1981.

[17] [17] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” Proc. of 2012 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 573-580, 2012. https://doi.org/10.1109/IROS.2012.6385773

FFD-SLAM: A Real-Time Visual SLAM Toward Dynamic Scenes with Semantic and Optical Flow Information

Hao Zhang*, Yu Wang**, Tianjie Zhong*, Fangyan Dong*, and Kewei Chen*,†

Hao Zhang^*, Yu Wang^**, Tianjie Zhong^, Fangyan Dong^, and Kewei Chen^*,†