2.1. Weeding Robots

Weeding mechanisms can be classified into three types: chemical, thermal, and mechanical. As examples of chemical-type mechanisms, the ARA Field Sprayer (Ecorobotix Co.) identifies weeds between crop rows using cameras and sprays herbicides accordingly ^c, whereas the See & Spray system (John Deere Co.) optimizes the spray area by controlling the nozzles ^d. The LaserWeeder G2 (Carbon Robotics Co.) employs a thermal-type mechanism that irradiates weeds using a high-powered submillimeter-precision laser ^e. Examples of mechanical mechanisms include the Rover (ROMI Co.), which suppresses weed establishment through periodic shallow tillage ^f. In addition, Iwano et al. developed a compact weeding robot equipped with a flail-type mowing mechanism ¹². Chemical-type systems raise concerns regarding their environmental impact, whereas thermal-type systems pose safety challenges. Considering both pedestrian safety and environmental impacts, mechanical weeding mechanisms are best suited for roadside environments.

Weeding robots equipped with mechanical weeding mechanisms have primarily been developed for agricultural and horticultural applications. The Titan weeding robot (FarmWise Co.) ^a targets only weeds within crop rows through visual processing, whereas the FD20 weeding robot (FarmDroid Co.) ^g enables precise weeding by memorizing seeding positions. Although large-scale weeding robots offer high operational efficiency, their size and weight make their operation difficult in roadside environments where drivable areas are limited. Compact weeding robots, such as the Automower (Husqvarna Co.) ^b and Landroid (Worx Co.) ^h weeding robots, achieve stable weeding operations using boundary line guidance and simple obstacle detection. However, because they assume known soil environments, they are not well-suited for roadside conditions that require weeding in unknown environments and on asphalt surfaces. In addition, compact robots face difficulties in continuous long-term operation due to battery capacity limitations, making a minimal sensor configuration and an efficient weeding mechanism design crucial.

2.2. Autonomous Navigation

In autonomous navigation, it is common to integrate multiple sensors to perceive the navigation environment. He et al. ¹³ combined light detection and ranging (LiDAR), inertial measurement unit (IMU), and global navigation satellite system (GNSS) to achieve robust and highly accurate self-localization even under disturbances in urban environments. Yuan et al. ¹⁴ constructed a navigation system for indoor service robots by integrating simultaneous localization and mapping (SLAM) using an RGB-D camera with landmark recognition. Shan et al. ¹⁵ integrated LiDAR with an RGB camera and IMU to achieve real-time map generation and self-localization. However, the LiDAR and RGB-D sensors are expensive, which increase the overall robotic costs.

To reduce costs, navigation methods using only a monocular RGB camera have also been developed. Classical image-processing-based methods include stereo vision ¹⁶ and techniques that combine inverse perspective mapping with RANSAC ¹⁷ to detect lane markings and determine travel paths. However, lane markings tend to fade because of road deterioration and are sensitive to lighting and surface conditions, limiting their robustness in real environments. Alternatively, deep learning models have also been developed to estimate travel paths directly. Bojarski et al. ¹⁸ controlled steering angles based on camera images using a convolutional neural network, whereas Badrinarayanan et al. ¹⁹ achieved high-accuracy extraction of drivable areas through semantic segmentation. Nevertheless, real-time processing can be challenging on edge devices, where latency and power consumption are critical. Moreover, these methods require large-scale training datasets and computational resources, making adaptation to unknown environments difficult.

2.3. Weed Detection Algorithm

The YOLO series is widely used as object detection models that balance real-time performance with detection accuracy. YOLO models estimate the object regions and class probabilities simultaneously, allowing them to operate on edge devices with limited computational resources. Among them, YOLOv8 ⁱ, which simplifies the inference mechanism through an anchor-free head, achieves a higher detection accuracy while maintaining lightweight performance. Recently, specialized models for weed detection in agricultural fields, such as YOLO-WDNet ²⁰ and CSCW-YOLOv7 ²¹, have been proposed. Although existing weed detection models are trained on top-view images, weeds along roadsides can only be captured from a lateral perspective. Moreover, considering the inference constraints of edge devices, reducing computational time and power consumption remains a significant challenge.

To specialize an object detection model for weed detection, fine-tuning with a weed image dataset is required. Publicly available research datasets include DeepWeeds ²², which contains eight wild weed species, and CWFID ²³, which provides annotated images of weeds in cultivated fields. However, these datasets do not include images captured in roadside environments where asphalt and other heterogeneous backgrounds are present. Therefore, it is necessary to capture images of weeds along roadsides and construct a dedicated dataset. Furthermore, if the dataset is built only for a specific roadside environment, the generalization ability of the weed detection model may be reduced. To ensure robust performance across various environments, a large-scale dataset is required; however, the manual annotation of such data is costly.

2.4. Positioning of This Study

We have previously developed a compact autonomous weeding robot specialized for roadside environments ²⁴. The developed three-wheeled robot measured 510 mm \(\times\) 670 mm \(\times\) 220 mm, weighed 11.6 kg, and was designed to operate without obstructing traffic, even on narrow roadsides. In the autonomous navigation system, the boundary line between the road and shoulder was detected using an RGB camera mounted in front of the robot as a robust feature against road surface degradation. Subsequently, a semantic segmentation model was developed to identify drivable areas, and a travel path was generated along the boundary line to achieve low-cost autonomous navigation without relying on GPS or LiDAR. In addition, an RGB-D camera mounted on the side captured the roadside area, and a weed-detection algorithm based on YOLOv8 was implemented to extract the weed regions. Based on the detected distance of the weeds, a nylon-cord weeding mechanism was deployed using an extendable arm, enabling safe and energy-efficient weeding operations.

Although our previous system demonstrated the fundamental functions of a roadside weeding robot, there are several issues that must be addressed for practical deployment. First, owing to imbalances in the center of gravity and the absence of an outer cover, autonomous navigation may become unstable. Second, navigation performance has not been sufficiently verified for diverse road geometries and long-distance travel. Third, the weed detection algorithm performs inference for weeding areas even in weed-free areas, resulting in additional computational time. Furthermore, the small size of the training dataset makes the model highly dependent on the specific image acquisition environment. Fourth, weeding tasks in real environments have only been verified in limited test sections.

In this study, we propose a practical roadside weeding robot that addresses these challenges. The hardware design is compact and durable for long-term outdoor operations. To ensure autonomous navigation, we evaluate the robot’s stability under complex curved paths indoors and verify its long-distance performance outdoors. For weed detection, we introduce a two-stage algorithm. The first stage performs lightweight vegetation presence classification, and the second stage detects weed regions using a lightweight version of YOLO, thereby reducing computational time. In addition, we construct a pseudo-weed dataset to enhance generalization performance. Finally, we conduct real-world experiments on autonomous navigation and roadside weeding tasks in outdoor environments.

3.1. Hardware Configuration

Figure 1 shows the external appearance and internal structure of the proposed robot, and Fig. 2 illustrates its hardware configuration. The proposed robot has a width, depth, and height of 603, 512, and 368 mm, respectively, with a total weight of 16.4 kg. The robot has a three-wheel configuration consisting of two front drive wheels and a rear caster for maneuverability. The frame is made of lightweight yet rigid aluminum, achieving both durability in outdoor environments and ease of handling. An RGB camera is used for path planning during navigation. The camera is mounted on the upper front of the robot to provide a wide field of view in the forward direction while minimizing the robot’s shadow and interference from the weeding mechanism. An RGB-D camera installed on the left side of the robot is used for weed detection. To acquire positional information about the target weeds, the RGB-D camera is positioned slightly ahead of the weeding mechanism along the traveling direction, allowing sufficient processing time between detection and weeding. The weeding mechanism is mounted on the left side of the robot behind the RGB-D camera, enabling accurate alignment with the weeds identified by the camera. A LiPo battery is employed as the power source and is positioned at the center of the robot to maintain its stability during movement.

The onboard computer functions as the main processing unit of the robot, handling image processing for weed detection and path planning based on the camera images. It also controls the weeding mechanism and drives the wheels based on the processing results. Because image processing involves machine learning, a high computational performance is required. Simultaneously, reducing the overall weight of the system is important to improve mobility efficiency. To satisfy both requirements, we employ the Jetson Orin Nano (NVIDIA Co.), which features a high-performance GPU for an edge device and enables real-time processing while maintaining low power consumption.

For the mechanical weeding mechanism of the proposed robot, we employ the STG302 motor (Greenworks Co.) as the cutting unit. The motor is rotated at a no-load speed of 9000 rpm, and a nylon cord is attached to its spindle. The centrifugal force generated by the rotation is used to cut the weeds. Compared with metal blades, the nylon cord is lighter and poses a lower risk of kickback, providing greater safety for outdoor operations. In addition, the weeding mechanism is mounted at the tip of an extendable arm that can elongate up to 30 cm according to the position of the detected weeds. Fig. 3 shows the states of the weeding unit during operation and under idle conditions. A linear rail is used for the arm’s motion system, which allows the mechanism to reach weeds located slightly away from the robot’s body while reducing the risk of accidental contact when inactive. The operation of the weeding unit is controlled based on the weed detection results. Specifically, the weeding process begins when the side-mounted camera detects weeds continuously for three consecutive frames. The arm is extended according to the distance to the roadside edge, and the nylon cord is rotated to reach a sufficient cutting speed. Conversely, if no weeds are detected in 10 consecutive frames, the weeding motor decelerates and the arm retracts. This procedure prevents unnecessary interruptions caused by temporary detection losses and helps maintain the operational efficiency. Through this control flow, the system can operate the weeding mechanism efficiently and safely according to the spatial distribution of the weeds.

The proposed robot is equipped with two cameras as the visual sensors. Fig. 4 shows examples of the images captured by these cameras. First, the forward-facing RGB camera (WEBCAM-A001; Tinzzi Co.) is used to recognize the road conditions to determine the travel path. For real-time autonomous navigation, the frame rate is set to 5 fps and the image resolution is \(640 \times 480\) pixels. The RGB-D camera (RealSense Depth Camera D405; Intel Co.) mounted on the left side of the robot is used to detect weed regions and measure the distances. This camera is operated at 10 fps with an image resolution of \(640 \times 360\) pixels. The depth information is obtained based on disparity, and a spatial filter is applied to the resulting depth image to smooth it while preserving the edges. For roadside distance estimation, YOLO is first used to detect the bounding boxes corresponding to the weed class from the RGB image. Assuming that the roadside exists in the background behind the weeds, we obtain depth values within the 10% height band around the center of the bounding box. Because the depth sensors are prone to missing data and outliers in the foreground, the representative depth value is determined by considering the top 5% of the largest depth values within the band. This approach enables a stable estimation of the distance to the curb in the background.

3.2. Autonomous Navigation

Figure 5 shows the block diagram of the autonomous navigation system of the proposed robot. First, the RGB camera mounted in front of the robot is used to detect the drivable road area. The left edge of the detected region is treated as the road boundary line. The boundary line points are then projected from the image coordinates onto the robot coordinates, and a travel path is generated by offsetting the boundary in the normal direction by a fixed distance. In addition, the rotational velocity obtained from the wheel encoders is used to estimate the robot’s heading direction and posture along the generated path. Finally, the motors are controlled using the Pure Pursuit algorithm ²⁵ such that the robot follows the target path.

In this study, fast segment anything model (FastSAM) ²⁶ is used to extract the road regions from the camera images via semantic segmentation. FastSAM combines the YOLOv8 architecture with a promptable segmentation function of the segment anything model, thereby enabling real-time segmentation. Fig. 6 shows an example of the segmented image. Because the RGB camera is mounted at the center of the robot, the region containing pixels in the lower central area of the image is regarded as the road region. To reduce noise, only the structure with the largest connected components is extracted after the dilation and erosion processing. In this study, we assume that the left edge of the road region corresponds to the road shoulder, and thus, the leftmost edge is treated as the set of boundary line points between the road and shoulder.

Because the boundary line points exist in the image coordinate system, they must be converted into a robot coordinate system. In this study, the ground plane is defined as \(z=0\), and the robot coordinate system is defined as an orthogonal coordinate system with its origin located at the midpoint between the two wheels of the robot. The camera coordinates \((X_C, Y_C, Z_C)\) for a pixel \((u, v)\) are given by

Following the obtained travel path, the driving mechanism is controlled using the Pure Pursuit algorithm. The Pure Pursuit algorithm is a control method that sets a look-ahead point along the target path at a fixed distance and determines the steering angle required to pass through that point. In this study, the look-ahead distance was empirically set to 0.4 m. This value was selected by considering the trade-off between the tracking stability and responsiveness during curved-path navigation. Segmentation inference can only be performed intermittently when the robot moves continuously. Therefore, the rotational speed is acquired from both wheel encoders, and the robot’s position and orientation are estimated continuously using odometry. When the segmentation inference is complete, the robot’s position and orientation are estimated, and the control is corrected based on them to ensure continuous travel without deviation from the travel path.

3.3. Weed Detection Algorithm

In this study, we propose a lightweight and high-speed weed detection algorithm. In the first stage, a lightweight binary classification gate is used to determine the presence or absence of weeds and distribute the computational processing load. The system proceeds to the second stage only when weeds are detected, and a lightweight YOLO model estimates the position and confidence scores of the detected weeds. When no weeds are detected, the subsequent object detection stage is skipped, thereby reducing the computational cost. Fig. 7 shows the block diagram of the proposed method. First, the image captured by the side-mounted camera is divided into four sections. A 10% overlap margin is applied between adjacent regions to avoid missing weeds near the division boundaries. In this study, only two right-segmented images are used, excluding the weeding mechanism and its shadow. The two-stage detection process is first applied to the left segment near the weeding mechanism. If no weeds are detected, the same process is applied to the right segment. This structure reduces the computational cost in weed-free areas and enables real-time processing on the edge devices.

For the first-stage weed identification, a lightweight binary classifier based on MobileNetV3 ²⁷ is adopted. MobileNetV3 features an efficient architecture using depthwise separable convolutions and a hard-swish activation function, making it suitable for fast inference on edge devices. In this study, we replace the final classification layer of a pretrained MobileNetV3-Small (trained on ImageNet ²⁸) with a two-class classifier: “Weed” and “None.” The input images are resized to \(224 \times 224\) pixels, and the class probabilities are output using a softmax function. The training dataset consists of 480 images with weeds and 480 images without weeds, all of which are captured using the side-mounted camera. During training, only the classifier layer is fine-tuned at a learning rate of \(3 \times 10^{-4}\). The maximum number of epochs is set to 15, and the model from the epoch that yields the best validation performance is selected.

For the second-stage object detection, YOLO11 ²⁹ is adopted to achieve a balance between small-object detection performance and real-time operation. However, to ensure efficient inference on edge devices, the network structure is partially modified to reduce its size. In the proposed model, the P5 layer, which mainly contributes to large-object detection, replaces its standard convolutional layers with GhostConv ³⁰ and reduces the number of output channels from 1024 to 768. GhostConv first generates a small number of feature maps using a standard convolutional layer and then applies low-cost transformations to increase the correlation between these maps. This enables efficient model size reduction while maintaining accuracy in the P5 region with a low spatial resolution. For the P4 layer and subsequent layers, which are crucial for medium- and small-object detection, the block structure and channel widths remain unchanged. To further improve the processing speed, the input image size is reduced to \(320 \times 320\) pixels.

The object detection model is trained using a dataset composed of real and synthetic images. Fig. 8 shows examples of the real and synthetic images. The real dataset includes 299 images of roadside scenes with and without weeds. Each image is manually annotated with rectangular bounding boxes around the weed regions, excluding off-road vegetation and background noise. However, because real images alone do not provide sufficient diversity in weed appearance or background conditions, 580 synthetic images are added. To generate these synthetic images, the GrabCut algorithm ³¹ is first used to remove background components from the real weed images, producing foreground images containing only weeds. Images of the roadside environments without weeds are used as the background images. To enhance robustness, the background dataset includes challenging elements such as drainage channels and pavement joints, which may induce false detections. The foreground weed images are then pasted onto the background images along the roadside boundary line, and random scaling is applied to create diverse combinations. Finally, the model is trained in two stages: pretraining on the synthetic images, followed by fine-tuning on the real images. This two-stage training strategy enables the construction of a generalizable and robust object detection model capable of operating reliably in unseen outdoor environments, even with limited computational resources. The initial training is executed for 200 epochs, followed by 100 epochs of fine-tuning on real data using the Adam optimizer. The model from the epoch with the best validation performance is selected as the final version.

4.1. Autonomous Navigation System

A series of traveling experiments was conducted in indoor environments to evaluate the autonomous navigation performance of the proposed robot. The proposed robot was intended to operate on the roadsides in Japan. According to the Japanese road design standards, the curvature of typical road alignments usually remains below 0.1. However, roadside weeds can also grow along more complex roadside geometries with higher curvatures. If the robot can autonomously navigate in such environments, its applicability to a wider range of roadside weeding tasks can be demonstrated. Therefore, indoor experiments were designed to evaluate the navigation performance of the proposed robot for roadside geometries with curvatures greater than 0.1.

As shown in Fig. 9, three types of course maps were prepared by arranging blocks with a height, length, and width of 19, 39, and 10 cm, respectively. Indoor environment A consisted of a course with a right-hand curve, indoor environment B included a left-hand curve, and indoor environment C featured a composite course that combined right-hand and left-hand curves. For each course pattern, the robot performed five test runs with its initial position adjusted to align with the wall before each trial. During the experiments, a distance sensor was mounted on the left side of the robot to measure its lateral distance from the wall during autonomous navigation. The travel speed of the robot was set to 0.25 m/s.

Indoor experiments were conducted on a carpet-like floor surface. Because this surface differs from real roadside surfaces (such as asphalt or concrete) in terms of friction characteristics and surface conditions, the results should not be interpreted as a precise reproduction of real road surface conditions. The purpose of the indoor experiments was to examine the basic navigation behavior of the proposed system when following boundary geometries including curved sections.

Figure 10 shows the distance error relative to the target distance. The mean error over five runs was calculated for each course pattern. In indoor environment A, which included a right-hand curve, the robot tended to drift outward as it entered the curve, thereby increasing the distance error from the block. Similarly, in indoor environment B, which included a left-hand curve, the robot temporarily approached the block as it lost sight of the block during sharp turns. However, in the straight section, from the start of the experiment to 20 s after the experiment commenced in indoor environment B, the robot maintained a stable trajectory while continuously adjusting its distance from the wall. Furthermore, after 37 s in indoor environment A, and after 42 s in indoor environment B, compensatory behavior was observed as the robot attempted to correct the accumulated error after passing the curves. In indoor environment C, the distance errors increased at 21 s (right-hand curve entry) and 58 s (left-hand curve entry). Overall, the robot successfully achieved stable autonomous navigation throughout the course. Although an increase in temporary error was observed during curve entry, the results confirmed that the robot maintained stable traveling performance on straight sections.

4.2. Weed Detection Algorithm

We verified the effectiveness of the proposed weed detection algorithm. As a baseline, a YOLO11n model pretrained on the COCO dataset ³² was fine-tuned using real-world data. The evaluation metrics were mAP@0.5, recall at a YOLO confidence threshold of 0.25, number of model parameters (M), and processing time [ms]. The processing time was measured as both the average and minimum values using the Jetson Orin Nano computer. The input image size for the YOLO model was set to \(320 \times 320\) pixels.

Table 1 shows the experimental results. The baseline model (YOLO11) was compared with three variants: one trained with synthetic images (Synthetic), one using a lightweight version of YOLO (Lightweight), and one incorporating a lightweight weed classification module (Gate). The introduction of synthetic data improved the recall from 0.79 to 0.81, demonstrating that the number of missed detections of weeds was reduced. The lightweight YOLO model decreased the mAP and recall to 0.70 and 0.77, respectively; however, the model reduced the number of parameters by 30%. The model with the weed classification module decreased the mAP and recall to 0.60 and 0.71, respectively; however, the average inference time was significantly reduced to 18.5 ms, with a minimum inference time of 5.3 ms. Due to the two-stage inference structure, the number of YOLO calls was reduced to 23% of the total, suggesting a trade-off between inference performance and computational time. The ablation experiments confirmed that, although the mAP decreased compared with the baseline, the introduction of synthetic data improved the recall and the addition of the weed classification module significantly reduced the inference time.

4.3. Field Experiments in Outdoor Environments

First, we evaluated the performance of autonomous navigation in real-world environments. As shown in Fig. 11, the experiments were conducted on paved roads surrounding two university campuses.

Outdoor environment A was characterized by fallen leaves and vegetation, making it difficult to distinguish the boundary line between the roadside and pavement. The experiments were conducted in this environment during daytime. In contrast, outdoor environment B had a clean roadside. However, for route H, which was illuminated by streetlights, the experiments were conducted after sunset to evaluate the performance under low-light conditions. During all experiments, the travel speed was set to 0.50 m/s, and the distance error relative to the roadside was measured.

Before discussing the tracking error, we evaluated the recognition stability of the road region in outdoor environments. Since the proposed method generates the travel path based on the left edge of the segmented road region, stable segmentation is important for stable boundary extraction and path generation. Therefore, the temporal variation of the segmented road area ratio was analyzed as a simple indicator of segmentation stability. Figs. 12 and 13 show the segmented road area ratios in outdoor environments A and B, respectively, including all navigation routes in each environment. The results indicated that the segmented road area ratio tended to be more stable in outdoor environment B compared with in outdoor environment A, which was consistent with the clearer roadside boundary in outdoor environment B. Comparison with the distance error results suggests that, particularly in outdoor environment B, the tracking error increased at around the same time as the changes in the segmented road area ratio. These results indicated that segmentation stability affected boundary extraction and path generation, and that the proposed method achieved sufficiently stable autonomous navigation under the tested outdoor conditions.

Figures 14 and 15 show the distance errors for environments A and B, respectively. In environment A, where the roadside was indistinct and it was difficult to determine a clear boundary line, the distance error tended to fluctuate. Nevertheless, the overall error remained within 10 cm, indicating successful tracking of the roadside boundary line. In environment B, the autonomous navigation was stable throughout, and the distance error remained within 5 cm for most sections. Stable autonomous navigation was also achieved under streetlight illumination after sunset. These results confirmed that, regardless of the roadside conditions or lighting environments, the proposed robot maintained stable autonomous navigation at the target velocity, maintaining a distance error of 10 cm.

Although the overall tracking performance in outdoor environment B remained stable, large distance errors were observed locally. The segmentation results for these intervals are shown in Fig. 16. During these intervals, the curb temporarily disappeared from view, and the segmentation became unreliable, resulting in incorrect estimates of the road boundary. This shifted the extracted boundary line and generated an incorrect travel path. Because the deviations were observed only over a short interval and the tracking recovered afterward, this error was interpreted as a local and temporary recognition failure rather than a persistent limitation of the proposed navigation method.

Subsequently, we evaluated the weeding performance of the proposed robot in two roadside environments. In roadside environment A, which consisted of a 4.90 m straight roadside with 18 patches of vegetation, the weeding performance was tested in a single operation. Fig. 17 shows the weeding process in roadside environment A. The weeds that had grown before the operation were cut properly after the robot passed over them. Consequently, more than 90% of the weeds were removed, demonstrating effective weeding during autonomous navigation. The dense clusters of weeds appearing at 14 and 38 s were also successfully removed, indicating that sufficient weeding could be achieved in a single pass. Roadside environment B consisted of a 5.30 m straight roadside fully covered with vegetation. Because one pass was insufficient to complete the weeding, the robot performed three passes along the same route. Fig. 18 shows the weeding process in roadside environment B. After the first pass, weeds still remained throughout the area, suggesting that the thin nylon cord alone was insufficient to cut through dense weeds. During the second and third passes, proper weeding was observed in several sections, particularly in the first half of the path. This demonstrated that the proposed robot could adapt to densely vegetated areas through multiple passes. However, weeds remained in the latter sections, which was likely because of nylon cord wear during earlier operations, preventing the cord tips from reaching the weed bases. Overall, these results confirmed that the proposed robot effectively performed autonomous weeding in real environments and could adapt to varying weed densities by conducting one or more passes as necessary.