1. Introduction

In recent years, teleoperation robots and autonomous mobile robots in real-world environments where humans live have attracted attention, and research and development in this field have been actively conducted. Competitions such as the Tsukuba Challenge ^a and the Nakanoshima Robot Challenge ^b provide opportunities for demonstration experiments in urban environments to accelerate research and development. Most teams participating in these competitions adopt LiDAR-SLAM [1–3] as their navigation system. However, many teams fail to complete the course. One of the main reasons for this is that the robot loses its self-position when there are significant differences between the point cloud data obtained during driving and the pre-created point cloud map. Additionally, in open environments where the point cloud is sparse, the lack of information can also cause the robot to lose its self-position.

Humans can easily reach destinations even in unfamiliar environments. Although the oft-cited “80%” figure for visual input lacks definitive empirical proof ⁴, little evidence contradicts vision’s predominance among the senses. We therefore treat visual information as the primary cue for human locomotion. In this paper, we propose a navigation method based on image information instead of conventional robot navigation using point cloud data. We conducted an investigation to observe what humans focus on while walking in real world and when remotely operating a robot using VR. In VR, it is also possible to remove surrounding information other than the road and present it to the subjects. The results indicated that if the recognition of the road is possible, it may be feasible to complete a course without a pre-existing map.

Considering the above, we have developed a simple navigation system that uses vanishing points to recognize only roads and does not require a map. While previous studies using this method have reported only short distances in a straight line, our study demonstrated that, over several hundred meters, a robot can successfully continue following a road trajectory and complete navigation using only visual information. Although the applicability does not extend to environments with complex junctions such as intersections or plazas, the proposed method exhibits advantages, including low preparation cost, system simplicity, and ease of recovery, when limited to road-following scenarios. As a complementary solution to LiDAR-based SLAM, our approach serves as an effective means to expand the range of options in situations where low-cost and low-preparation deployment is required.

2. Environmental Recognition Investigation During Human Walking

First, ethics issues are explained as follows. Sections 2 and 3 of this paper describe an investigation into human walking behavior. Only fully anonymized data, excluding any personally identifiable information (such as names, addresses, or contact details), were used. According to the “Checklist for Ethical Review of Research Involving Human Subjects at Kansai University,” it was confirmed that this study does not fall under the category of “research involving human subjects” (as defined in Article 2 of the Kansai University Ethical Regulations) and therefore does not require ethical review.

To investigate environmental recognition during human walking, the subjects were asked to walk within the university campus. During the walk, a wearable device with capabilities of simultaneously recording image and audio (THINKLET, Fairy Devices) was mounted at the subjects’ eye level (Fig. 1). The subjects were asked to freely respond to two questions during their walk: “What are you consciously looking at while walking?” and “What are you consciously thinking about while walking?” (Table 1(a)). The subjects were five males, two aged 22, one aged 23, and two aged 24. Although no eye tracking devices or high-precision GPS devices were used in this experiment, reviewing the recorded images and audio after the walk allowed for a certain degree of quantitative verification of where the subjects looked and walked.

The investigation results are shown in Table 1(b) and Fig. 2. Although various responses were obtained, the most common response was that the subjects were consciously aware of the road while walking. Based on this finding, a further investigation was conducted to determine which parts of the road the subjects focused on and where they were looking (Table 2(a)). The results are presented in Table 2(b) and Fig. 3. The findings revealed that the subjects identified the boundaries of the road by paying attention to curbstones, steps, and physical obstacles such as buildings and fences. The major road features identified in the survey are illustrated in the photographs in Fig. 4.

In summary, the results of this survey revealed that humans tend to focus on the road surface, dynamic obstacles, and the surrounding environment while walking, often looking into the distance. Among these, particular attention was given to the road surface, with the subjects being especially aware of the road area and road boundary. Based on this, the hypothesis was formed that humans might be able to walk using only road area information. To further investigate human environmental recognition, an additional survey using VR was conducted in the next section.

3. Investigation of Human Environmental Recognition During Robot Teleoperation Using VR

We investigated the environmental recognition humans perform when remotely controlling a robot using VR. The robot was operated from the perspective of a camera installed on the robot. Based on the results obtained in the previous section, we also presented only the road information in VR by removing information of the surroundings.

A research and development model of an electric wheelchair (WHILL Model CR, WHILL Inc.) was used as the mobile robot equipped with sensors, a portable power supply, and a notebook PC for control (Fig. 5(a)). A stereo camera (ZED2i, Stereolabs Inc.) was used as the image sensor, and the RGB-D images obtained from it were utilized (Fig. 5(b)). The VR headset worn by the subjects (Meta Quest 2, Meta Inc.) is shown in Fig. 5(c).

An overview of the experimental method is shown in Fig. 6. The PC used to control the robot outdoors and the PC used by the subjects to view the VR images indoors are connected to the same wireless LAN available on the university campus. Through LAN communication, video and robot movement commands (for steering angles and speed) are sent and received, enabling remote control of the robot. The development of this software utilized ROS packages [c–e]. This experimental method not only ensures that the subjects can safely engage in the experiment but also provides the advantage of using robot-perspective video to investigate how humans process visual information.

The subjects wore VR headsets and conducted teleoperation experiments in two patterns: one using the standard video and the other using a video that displayed only the road area (Figs. 7(a) and (b)). After the experiment, a questionnaire was conducted asking about “whether the course could be completed using only the road area video,” and “the differences in difficulty between the two teleoperation patterns, as well as their honest impressions and observations” (Table 3(a)).

The survey results are shown in Table 3(b). All five subjects successfully completed the designated course using only the road area video, while avoiding obstacles. This demonstrates that, at a minimum, humans can remotely control a robot with just road information. However, there were gap in assumed vs. actual robot position, since landmarks were unavailable. Also, the corners were difficult to judge. It was found that the environmental information other than road was also useful for easy and secure human walking.

Based on the survey results in Table 3, it is believed that the visual information used by humans during remote control is categorized into levels based on their importance (Fig. 8). These levels are primarily divided into three categories, with the complexity of the information increasing as the level rises. Level 1 involves the recognition of the road area. As long as the road is visible, humans were able to control the robot remotely. Level 2 is the recognition of obstacles, which is directly related to safety during walking. In the remote control experiment, as soon as pedestrians’ feet or bicycles appeared in the video, the subjects instructed the robot to avoid them. Finally, Level 3 involves the completion of self-positioning using semantic information. Information about the surroundings, such as buildings and signs, is used to estimate the robot’s location and serves as a reference for determining self-position.

In the context of autonomous driving for the robot, it is believed that by relying on Level 1 road area information for navigation, using Level 2 obstacle information to avoid hazardous objects, and utilizing Level 3 environmental information to estimate self-position and follow the correct path, it is possible to realize robot navigation that mimics human environmental recognition.

4. Navigation with Image Information Based on Investigation Results

As described in Sections 2 and 3, humans can walk or remotely operate robots using only road area information (Level 1). It was also found that humans tend to gaze at the distant part of the road while walking (Table 1(b)). If the robot can navigate using only the road surface region, then in road-following environments, it is possible to achieve low-preparation-cost autonomous navigation without the need for prior construction or matching of dense point cloud maps, as required in conventional LiDAR-SLAM approaches.

Since the subjects were aware of the road area and its distance (vanishing point), we created a program to calculate the vanishing point of a road (Fig. 9) ^{5,6,7,8,9,10,11,12}. The software was implemented using the OpenCV package [f–i]. It detects the boundary lines of the road edges captured by the camera ^7,8,9,10,11 (Fig. 9(b)) and identifies the location with the highest intersection density of these lines as the road’s vanishing point ^7,12 (Fig. 9(c)). The robot’s navigation method is shown in Fig. 10. Software was designed to adjust the robot’s posture to ensure that the vanishing point remains on the centerline of the video frame. The actual navigation results are presented in Figs. 11 and 12.

On a gently curved road like the one shown in Fig. 11, the robot successfully maintained its position in the center of the road, preventing a collision with the hedge that would have occurred if it had continued straight. However, at the L-shaped intersection in Fig. 12, the camera failed to capture the curb or step, leading to unsuccessful line detection and vanishing point calculation, resulting in a failure of autonomous driving. To address this issue, an additional algorithm was implemented, enabling the robot to rotate in place at locations where road boundary detection is difficult, as shown in Fig. 12, until it successfully detects the road boundary and recalculates the vanishing point (Fig. 13). Once the road boundary and vanishing point are detected again, the robot resumes its navigation using the method described in Fig. 10. After implementing this algorithm, further autonomous driving experiments were conducted. As a result, the robot successfully navigated the L-shaped intersection (Fig. 14) and completed a drive along the roads surrounding the university campus (Fig. 15).

However, the proposed method is primarily designed for road-following, and the turning behavior at corner regions is only a supplementary mechanism, implemented in a simplified manner to maintain the intended direction of travel along the road. At present, the system does not implement autonomous decision-making for selecting a travel direction at intersections. The implementation of such a function remains a future challenge.

5. Evaluation of the Present Method

5.1. Performance of Following Road

This method is able to achieve autonomous driving without prior map creation. Conventional methods using LiDAR or GNSS require the creation of maps and route plans in advance. In the case of LiDAR, it is necessary to drive around the course beforehand to generate a point cloud map and create waypoints. In the case of GNSS, a detailed driving map must be prepared in advance by specifying the exact waypoint positions using satellite images from Google Maps, making the pre-driving preparation process time-consuming.

In our laboratory, self-localization has also been achieved using LiDAR- and GNSS-based methods, and the robot has successfully completed navigation around the campus facilities shown in Fig. 15 ^13,14. The respective driving results are presented in Figs. 16 and 17. On the other hand, the proposed method successfully completed a circular course on our campus with minimal preparation. This contrast does not indicate a general superiority or inferiority of the methods, but rather represents an evaluation limited to the specific perspective of preparation cost.

A previous study on autonomous driving using road vanishing points ¹⁵ successfully achieved driving on a 50-meter straight course. In contrast, the our present study demonstrated the adaptability of the proposed method to non-linear routes by successfully completing a course of approximately 350 meters that included straight sections, gentle and large curves, slopes, and sharp turns such as L-shaped intersections.

The deviation of the robot from the road center during autonomous driving was investigated. The approximately 350-meter course shown in Fig. 18 was divided into five sections based on road shape and surrounding environment. The robot performed autonomous driving by following the vanishing point, and its deviation from the road center was measured at 10-meter intervals. The road center was set as the coordinate origin (0), with negative values representing the robot’s position to the left of the center and positive values representing its position to the right. Additionally, defining the road width as 2\(L\) and the robot’s deviation from the center as \(x\), the error rate \(\varepsilon\) was calculated using the formula \(\varepsilon=\vert x\vert/L\times 100{\%}\). The measurement results are shown in Figs. 19 and 20. Table 4 summarizes the maximum, average, and standard deviation of the error rates for each section.

On relatively narrow roads with straight sections and gentle curves, like section 1, the error rate was low, and the robot maintained its position near the road center. On the other hand, in section 2 (L-shaped road) and section 4 (large curve), indicated in yellow in Fig. 18, the error rate was higher, and the robot deviated from the road center. This deviation occurred because the robot lost sight of the road boundary. To regain the road boundary, the robot subsequently performed a turning motion.

Additionally, higher error rates were observed in section 3 and section 5. This was because, after performing turning maneuvers in section 2 and section 4, the robot temporarily moved closer to the edge of the road. In section 5, however, the robot gradually returned to the center of the road after traveling a certain distance, resulting in a decrease in the error rate.

Based on these results, it was found that the proposed method, which follows the vanishing point obtained from the road boundary, tends to exhibit significant short-term deviations at corners, but it demonstrated good driving performance on straight road sections.

6. Conclusions and Future Guidelines

An environmental recognition study was conducted during human walking and robot teleoperation using VR. It was found that humans tend to gaze at the ground, dynamic obstacles, the surrounding environment, and distant areas while walking. In particular, they pay significant attention to the road surface, with the road boundaries serving as important visual information. Although this observation may seem intuitive, the significance lies in having performed the investigation and confirmed these findings through actual survey data.

Robot posture control was performed using the road’s vanishing point. As a result, the robot successfully navigated along the center of the road on straight sections and roads with gentle curves. In environments like L-shaped roads, where detecting the road boundary is difficult, the robot successfully searched for the road boundary by performing a turning maneuver. Experimental evaluation of the deviation from the road center on the university campus course showed that, while a significant deviation of over 70% was observed in curved sections, the error remained below 20% on straight sections. Compared to previous studies using vanishing point information, our method successfully navigated long distances that included curves. Under the experimental conditions of this study, advantages were observed in reducing prior preparation efforts, such as point cloud map construction and waypoint setting. On the other hand, the applicability of the proposed method is limited in environments with complex topologies, such as intersections and plazas, and it is therefore positioned as a complementary option to LiDAR-SLAM.

Although the proposed system can run the course, it cannot run in environments with many dynamic obstacles (humans, bicycles, vehicles, etc.) because it does not have a dynamic obstacle avoidance function. Future work includes adding an algorithm to recognize and avoid dynamic objects and developing a self-position estimation method using information on the surrounding environment.