2.2.1. Tier IV Camera

Spot was equipped with cameras on the front, rear, left, and right. However, these were designed to capture the ground obliquely downward to check the footing, making it difficult to sufficiently capture workers at a distance. Furthermore, since this system is intended for outdoor use, there is a problem that optical ghosting artifacts appear in the images under backlit conditions, reducing the accuracy of human tracking. In addition, because tracking is required even when workers move laterally, a camera with a wide field of view (FoV) is required. Therefore, we adopted a Tier IV C1-120 manufactured by Tier IV, which is also used in automotive camera applications ¹². The camera was connected to the Jetson Orin Nano through a GMSL2-USB3.0 converter, and the acquired images were processed on the Jetson Orin Nano. Fig. 3 shows the appearance of the Tier IV C1-120, and its intrinsic parameters were computed using MATLAB’s Computer Vision Toolbox. To ensure consistency with the input size of the detection model, calibration was performed using preprocessed images, and the original images (\(1920\times1280\)) were cropped by 100 pixels from both the top and bottom to \(1920 \times 1080\) pixels and subsequently resized to \(640\times360\) pixels. The resulting calibration parameters are listed in Table 2.

We also evaluated the installation angles of the cameras. The camera height (74 cm), bib height (170 cm), and distance (50 cm) yielded the required angle of approximately 62.5°. Considering a vertical FoV of 77° (\(\pm\)38.5°), an upward tilt of approximately \(62.5° - 38.5° = 24°\) ensured that the entire bib was captured.

Consequently, by installing the camera tilted approximately 24° upward, it was possible to capture the entire bib of the workers at a close distance within the FoV.

2.2.2. DJI MIC2

In this system, we adopted a DJI MIC2 to send commands to Spot via voice. With a Bluetooth connection, voice commands are sometimes not recognized correctly due to interference from other devices. Therefore, we connected the receiver to the Jetson Orin Nano via USB and sent voice commands via wireless communication with the transmitter. Although the specified communication distance was 250 m, the actual distance at which voice recognition functioned normally was approximately 75 m. Considering the size of the orchard, we considered this communication distance sufficient.

2.4.1. Detachable Mechanism for Harvest Baskets

A detachable attachment was installed on the EAP2 option to enable a one-touch replacement of the harvest basket. A bicycle detachment mechanism was adopted for a low-cost, reliable fixation and easy removal. While 20 kg baskets are common in the field, Spot’s payload constraints require the use of a smaller half-height basket. Fig. 6 shows the base and installation. This mechanism allows quick basket exchange after transport, thereby improving the continuity and efficiency of harvesting.

2.4.2. Electrically Actuated Basket for Discarded Fruits

An electrically actuated basket with an automatic opening and closing mechanism, originally developed in our previous work ¹³, was utilized to dispose of discarded fruits during thinning operations. Since thinned fruits have no commercial value, the design prioritizes lightweight construction, operational simplicity, and reliable discharge rather than careful handling.

Although the quadruped robot supports a maximum payload of approximately 14 kg, a portion of this capacity is consumed by the EAP2 module and onboard sensors (approximately 4 kg). Therefore, minimizing the mass of the basket and its actuation mechanism is essential for maximizing the remaining payload available for the transported fruits. To achieve this, an aluminum honeycomb structure was adopted for the basket frame, providing a high stiffness-to-weight ratio while maintaining sufficient mechanical strength under field conditions. Consequently, the developed basket achieves a total mass of 4.2 kg, enabling efficient utilization of the remaining payload capacity.

The opening and closing mechanisms were driven by a compact electric motor and the lid was controlled using GPIO signals from a Jetson Orin Nano mounted on the robot. Upon receiving a voice command to discard fruits, the quadruped robot autonomously moved to a predefined discarding location, opened the basket lid to discharge the contents, and returned to the working position. This parallelization of thinning and disposal tasks enables a single worker to continue thinning without interruption, thereby reducing the physical burden and improving overall work efficiency (Fig. 7).

3.1.1. YOLO-Based Bib Detection

For training and validation, 3099 images were collected at the Kochi University of Technology campus. During annotation, only the undeformed region of the bib (shoulder to hem) was labeled, excluding parts distorted by the wind. When a bib protruded beyond the image frame, it was annotated with a separate label (“hidden bib”) rather than the normal “bib” label, to prevent incorrect detection. During the operation, only bib labels were processed. From the dataset, 2479 images were used for training and 620 for validation. All images were resized to \(640\times360\), and training was conducted for 1000 epochs with a batch size of 16. A detection frame rate of 3.8 fps was achieved during the experimental trials. Examples of the detection results are shown in Fig. 8.

3.1.2. Distance Estimation and Tracking

The distance to the worker was estimated using a standard pinhole camera model. The relative position of the worker was determined within the image coordinate system using the intrinsic parameters of the camera calibrated in Section 2.2.1.

Let \(H_{\mathit{bib}}\) [mm] denote the actual vertical length of the bib in the physical world and let \(h_{\mathit{bib}}\) [pixels] denote the vertical length of the detected bib region in the image, measured as the height of the corresponding bounding box. The depth \(Z\) [mm] along the optical axis from the camera to the worker was computed using Eq. \(\eqref{eq:z_distance}\).

• If the bounding box center is located within 90 pixels of the left or right edges of the image, Spot performs only in-place rotation (without any forward motion) to prevent loss of the target from the FoV. The rotation angle is computed as \(\tan^{-1}(X/Z)\).

• When the bounding box center is near the center of the FoV and the estimated distance \(Z\) is within the range of 1.5–10 m, Spot approaches the worker by first rotating to align with the target and then moving forward.

• If the estimated distance \(Z\) is less than 1.5 m, Spot refrains from advancing forward to ensure safety. However, in-place rotation may still be performed to maintain the worker within the FoV.

• If the estimated distance \(Z\) exceeds 10 m, all motions are suspended, including both rotation and forward movements.

3.1.3. Navigation System

Spot maintains an environmental map and reproduces the path once walked ¹¹. The environmental map was created using LiDAR mounted on the EAP2 option and consisted of waypoints automatically created during walking, along with edges connecting those waypoints. This waypoint-edge graph is stored in the JSON format based on the GraphNav ^e, where each waypoint and edge includes the attributes listed in Tables 4 and 5. Waypoints are generated at the origin of the body coordinate system, that is, at the center of the body. Spot estimates its position using odometrical information. However, when it moves outside the mapped area or loses its location (e.g., after power-off), relocation is required. If fiducials, as shown in Fig. 10, are present on the map, Spot can re-estimate its position based on the onboard camera (front, rear, left, or right) that captures the fiducials and their apparent size. An example of a generated environmental map is shown in Fig. 11. Environmental maps can be created either by the bib-tracking method described above or by using the Autowalk function in Spot’s standard tablet application. During harvesting or thinning operations, it is important to register locations, such as trucks or disposal areas. After creating the map, issuing commands at these sites registers new waypoints, enabling Spot to navigate autonomously to them when instructed. When moving along an environmental map, the movement cost of an edge connecting two adjacent waypoints \(w_i\) and \(w_j\) is calculated as the Euclidean distance based on their 3D coordinates \((x_i, y_i, z_i)\) and \((x_j, y_j, z_j)\) using Eq. \(\eqref{eq:eq-cost}\).

Based on this cost, Dijkstra’s algorithm was applied to compute the shortest path from the current position to the target waypoint, allowing Spot to autonomously follow the optimal route.

4.4.1. Worker-Following Function

During field operations, Spot autonomously follows the worker, enabling continuous harvesting without manual repositioning. This reduces interruptions compared to manually operated transport vehicles and contributes to smoother task execution.

However, field trials revealed occasional tracking instabilities. Specifically, the robot temporarily lost detection when the identification bib was occluded by tree trunks or dense foliage or when the worker made abrupt directional changes between rows. Although tracking was typically reestablished once the worker reentered Spot’s FoV, these observations indicate that the current perception strategy remains sensitive to visual occlusion. Therefore, a more robust tracking and recovery method incorporating environmental mapping is required in future studies.

4.4.2. Voice Command Interface

The implementation of the voice command interface enabled hands-free operation of the robot even when workers’ hands were occupied, significantly improving the overall operability. Basic commands such as starting and stopping functions were successfully recognized in an actual orchard environment. With a clear line of sight, voice recognition was confirmed at distances of up to 75 m. The average command execution latency was 0.53 s from the onset of speech, indicating practical responsiveness to field operations. However, field trials also revealed that continuous conversations among workers occasionally interfere with the wake word detection or result in an increased processing latency. Therefore, the implementation of a more robust voice recognition method is required in future studies.

4.4.3. Detachable Mechanism for Harvest Baskets

The detachable mechanism, adapted from a bicycle attachment structure, enables the direct exchange of harvest baskets upon arrival at the transport vehicle. This eliminated the need to manually transfer the harvested fruits to another container.

The attachment and detachment processes required approximately 15 s and could be performed without specialized tools. This mechanism functioned reliably during repeated exchanges in field trials, demonstrating its practical suitability for orchard transport tasks. However, a practical limitation was identified in the physical design of the basket-attachment interface. The attachment to the bottom of the basket was approximately 3-cm-thick. When baskets filled with harvested fruits are stacked, this protruding component poses the risk of bruising or damaging the fruits in the underlying baskets. Therefore, the development of a detachable mechanism with thinner attachments to prevent fruit damage is required in future studies.