The digitization of the construction site environment has progressed rapidly. In this study, the operations of hydraulic excavators—which are machines widely used in the construction industry—were advanced to enable automation and unmanned operation. To achieve this, it is necessary to determine the environment of the machines at the sites, and a real-time measurement and visualization methodology that can be installed at common construction sites is required. In this study, we propose a measurement system for reconstructing a wide range of surrounding environments using machine-borne sensors mounted on a hydraulic excavator. The proposed system measures the entire surrounding environment using a sensor unit composed of a laser imaging detection and ranging (LiDAR) and a wide-angle camera. Furthermore, methods of time-series integration for wide-range and dynamic measurements during work for occlusion-robust visualization are proposed. In an experiment using actual machines on an earth-moving site, we validated the performance of our proposed system by quantitative evaluation and confirmed that the system provides an effective solution for the digitization of construction sites.

1. [1] R. Heikkilä, T. Makkonen, I. Niskanen, M. Immonen, M. Hiltunen, T. Kolli, and P. Tyni, “Development of an Earthmoving Machinery Autonomous Excavator Development Platform,” 36th Int. Symposium on Automation and Robotics in Construction, pp. 1005-1010, 2019. https://doi.org/10.22260/ISARC2019/0134
2. [2] S. Su, R. Y. Zhong, and Y. Jiang, “Digital twin and its applications in the construction industry: A state-of-art systematic review [version 1; peer review: 1 approved with reservations],” Digital Twin 2022, Vol.2, Article No.15, 2022. https://doi.org/10.12688/digitaltwin.17664.2
3. [3] O. M. U. Eraliev, K. Lee, D. Shin, and C. Lee, “Sensing, perception, decision, planning and action of autonomous excavators,” Automation in Construction, Vol.141, Article No.104428, 2022. https://doi.org/10.1016/j.autcon.2022.104428
4. [4] T. Nagano, R. Yajima, S. Hamasaki, K. Nagatani, A. Moro, H. Okamoto, G. Yamauchi, T. Hashimoto, A. Yamashita, and H. Asama, “Arbitrary Viewpoint Visualization for Teleoperated Hydraulic Excavators,” J. Robot. Mechatron., Vol.32, No.6, pp. 1233-1243, 2020. https://doi.org/10.20965/jrm.2020.p1233
5. [5] D. Yeom, H. Yoo, J. Kim, and Y. Kim, “Development of a vision-based machine guidance system for hydraulic excavators,” J. of Asian Architecture and Building Engineering, Vol.22, No.3, pp. 1564-1581, 2023. https://doi.org/10.1080/13467581.2022.2090365
6. [6] M. Wada and Y. Mori, “3D Digital Twin System for Remote Construction Work,” Hitachi Review, Vol.72, No.1, 2024.
7. [7] G. L. Besnerais, M. Sanfourche, and F. Champagnat, “Dense height map estimation from oblique aerial image sequences,” Computer Vision and Image Understanding, Vol.109, No.2, pp. 204-225, 2008. https://doi.org/10.1016/j.cviu.2007.07.003
8. [8] A. Lucieer, S. M. Jong, and D. Turner, “Mapping landslide displacements using Structure from Motion (SfM) and image correlation of multi-temporal UAV photography,” Progress in Physical Geography: Earth and Environment, Vol.38, No.1, pp. 97-116, 2013. https://doi.org/10.1177/0309133313515293
9. [9] S. Verykokou, A. Doulamis, G. Athanasiou, C. Ioannidis, and A. Amditis, “UAV-based 3D modelling of disaster scenes for urban search and rescue,” IEEE Int. Conf. on Imaging Systems and Techniques (IST), Chania, Greece, pp. 106-111, 2016. https://doi.org/10.1109/IST.2016.7738206
10. [10] Y. Shao, Y. Ji, H. Fujii, S. Yamamoto, T. Chiba, K. Chayama, Y. Tamura, K. Nagatani, A. Yamashita, and H. Asama, “Estimation of Soil Volume Change Using UAV-based 3D Terrain Mapping,” Proc. of the 2017 IEEE/SICE Int. Symposium on System Integration, pp. 247-250, 2017.
11. [11] S. Kundu, N. Das, D. Saha, and P. Biswas, “Unknown terrain imaging with adaptive spatial resolution using UAV,” Ad Hoc Networks, Vol.135, Article No.102937, 2022. https://doi.org/10.1016/j.adhoc.2022.102937
12. [12] H. Yoshida, T. Yoshimoto, D. Umino, and N. Mori, “Practical Full Automation of Excavation and Loading for Hydraulic Excavators in Indoor Environments,” 2021 IEEE 17th Int. Conf. on Automation Science and Engineering (CASE), pp. 2153-2160, 2021. https://doi.org/10.1109/CASE49439.2021.9551504
13. [13] M. Inagawa, T. Kawabe, and T. Takei, “Demonstration of Localization for Construction Vehicles Using 3D LiDARs Installed in the Field,” J. of Field Robotics, 2023 (in press). https://doi.org/10.1002/rob.22211
14. [14] M. Inagawa, T. Kawabe, T. Takei, and K. Nagatani, “Demonstration of Position Estimation for Multiple Construction Vehicles of Different Models by Using 3D LiDARs Installed in the Field,” ROBOMECH J., Vol.10, Article No.15, 2023. https://doi.org/10.1186/s40648-023-00252-0
15. [15] C. Sung and P. Y. Kim, “3D terrain reconstruction of construction sites using a stereo camera,” Automation in Construction, Vol.64, pp. 65-77, 2016. https://doi.org/10.1016/j.autcon.2015.12.022
16. [16] D. Yeom, H. Yoo, and Y. Kim, “3D surround local sensing system H/W for intelligent excavation robot (IES),” J. of Asian Architecture and Building Engineering, Vol.18, No.5, pp. 439-456, 2019. https://doi.org/10.1080/13467581.2019.1679148
17. [17] X. Li, C. Liu, J. Li, M. Baghdadi, and Y. Liu, “A Multi-Sensor Environmental Perception System for an Automatic Electric Shovel Platform,” Sensors, Vol.21, No.13, Article No.4355, 2021. https://doi.org/10.3390/s21134355
18. [18] A. Rasul, A. Khaicnour, and J. Seo, “Development of Integrative Methodologies for Effective Excavation Progress Monitoring,” Sensors, Vol.21, No.2, Article No.364, 2021. https://doi.org/10.3390/s21020364
19. [19] A. Rasul, A. Khaicnour, and J. Seo, “Effective Ground Mapping for Autonomous Excavation,” 2021 21st Int. Conf. on Control, Automation and Systems (ICCAS), pp. 1087-1092, 2021. https://doi.org/10.23919/ICCAS52745.2021.9650067
20. [20] I. Niskanen, M. Immonen, T. Makkonen, P. Keränen, P. Tyni, L. Hallman, M. Hiltunen, T. Kolli, Y. Louhisalmi, J. Kostamovaara, and R. Huikkilä, “4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator,” Automation in Construction, Vol.112, Article No.103112, 2020. https://doi.org/10.1016/j.autcon.2020.103112
21. [21] M. Immonen, I. Niskanen, L. Hallman, P. Keränen, M. Hiltunen, J. Kostamovaara, and R. Huikkilä, “Fusion of 4D Point Clouds From a 2D Profilometer and a 3D Lidar on an Excavator,” IEEE Sensors J., Vol.21, No.15, pp. 17200-17206, 2021. https://doi.org/10.1109/JSEN.2021.3078301
22. [22] I. Niskanen, M. Immonen, T. Makkonen, L. Hallman, M. Mikkonen, P. Keränen, J. Kostamovaara, and R. Huikkilä, “Trench visualisation from a semiautonomous excavator with a base grid map using a TOF 2D profilometer,” J. of Visualization, Vol.26, No.4, pp. 889-898, 2023. https://doi.org/10.1007/s12650-023-00908-4
23. [23] P. J. Besl and N. D. Mekay, “A method for registration of 3-D shapes,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.14, No.2, pp. 239-256, 1992. https://doi.org/10.1109/34.121791
24. [24] D. Sun, C. Ji, S. Jang, S. Lee, J. No, C. Han, J. Han, and M. Kang, “Analysis of the Position Recognition of the Bucket Tip According to the Motion Measurement Method of Excavator Boom, Stick and Bucket,” Sensors, Vol.20, No.10, Article No.2881, 2020. https://doi.org/10.3390/s20102881
25. [25] Z. Pučko, N. Šuman, and D. Rebolj, “Automated continuous construction progress monitoring using multiple workplace real time 3D scans,” Advanced Engineering Informatics, Vol.38, pp. 27-40, 2018. https://doi.org/10.1016/j.aei.2018.06.001