2.1. Snake-Like Robot

An overview of the proposed snake-like robot is shown in Fig. 1. Other than the body link with an inertial measurement unit (IMU) and a single-board computer (SBC), the structure is the same as the previous research ^23,27. The authors proposed a snake-like robot that can switch between undulatory locomotion and rolling locomotion by center of gravity shift. The rolling locomotion is generated by center of gravity shift using the deformation of multiple links, and it is a mode to realize high running efficiency on even ground. Undulatory locomotion such as sidewinding ^31,32 and swimming ^33,34 have high running performance on uneven terrain and in fluids, but they consume a lot of power.

The specifications of the snake-like robot are shown in Table 1. The total length of the snake-like robot is 1.76 m, the mass is 5.76 kg, and it is a multi-link mechanism in which the axes of 16 actuators (B3M-SC-1170-A, KondoKagaku, Co., Ltd., Japan) are arranged so that they are perpendicular to each other (Fig. 1(a)). It has the same specifications as the previous research ^23,27 except for the changes that the total length of the body link is increased by 100 mm and the mass is increased by 158 g. By covering the robot with a waterproof cover made of rubber and nylon, it maintains its waterproof and dustproof properties (Fig. 1(b)). By bending the joints in the dorsal and lateral directions, this robot performs undulatory locomotion like a snake.

Figure 1(c) shows the configuration in rolling locomotion. The shape of the parallel two wheel vehicle is achieved by deforming the head and tail links to the shape of the tire. The head and tail are designed to be on the same side when in the tire shape. This causes the head and tail to be directly below when deformed to the tire shape because they are more massive than the other links.

2.2. Modelling in Physics Simulator

A numerical model based on the multi-body dynamics formulation is required to design a controller for rolling locomotion by the deep reinforcement learning algorithm. In this study, the physics engine Genesis (v0.2.1) was used to model the rolling locomotion by the multi-body dynamics formulation ^a. Table 2 shows the parameters used in the numerical analysis of the multi-link rigid body model. The physical parameters such as mass and moment of inertia of the snake-like robot shown in Fig. 1 are obtained from the 3D model. However, modeling all links as movable makes the degrees of freedom too high and learning difficult, so links other than those necessary for the rolling locomotion are modeled as rigid bodies as shown in Fig. 2(a). The subscripts in Table 2 correspond to the numbers in Fig. 2(a). Among these, the rigid body ① consists of multiple links, and it is assumed that this shape is maintained by the actuators. The links from ② to ④ are the normal links of the snake-like robot, and they are driven to move the link ⑤ greatly. Link ⑤ has a larger mass than the other links as shown in Table 2. By moving this link, the center of gravity moves and the rolling locomotion occurs. The parameters of numerical time integration and the controller of the actuators are shown in Table 3.

For motion analysis, the coordinate systems shown in Fig. 2 are defined. \(O_i\) (\(i \in \{1, 2, 3, 4, 5\}\)) are the coordinate systems representing the center of gravity position of each link. When each joint bends at an angle of \(\phi_i\) as shown in Fig. 2(b), the position of \(O_i\) also changes. The moments of inertia shown in Table 2 are defined with respect to these coordinate systems. To evaluate the motion of the robot, a local coordinate system fixed to the body link (the middle link in Fig. 1(a)) that changes according to the motion of the body is defined as the body coordinate system (\(O'\)-\(x'y'z'\)). In the body coordinate system, the \(x'\)-axis is defined as the longitudinal direction of the body link (the direction of the body-axis), the \(z'\)-axis is defined as the normal direction to the upper surface of the body link, and the \(y'\)-axis is defined as the axis orthogonal to them. As the posture of the body changes, the orientation of each axis of the body coordinate system as seen from the world coordinate system \(O\)-\(XYZ\) also changes according to the amount of rotation of the body. The position and posture of the robot are described by the position vector \(\boldsymbol{r}_{O'}(t)\) of the origin of the body coordinate system as seen from the world coordinate system and a quaternion.

3.1. Action Space

The action is defined as in the previous research ^28,30. The angle range and angular velocity of the actuators have physical limitations. To satisfy the driving angle range constraint \(\phi_{\mathrm{min},l} \leq \phi_{\mathrm{d},l}(t) \leq \phi_{\mathrm{max},l}\) and the angular velocity constraint \(\left| \mathrm{d} _t \phi_{\mathrm{d},l}(t)\right| \leq \omega_\mathrm{max}, \: l \in \{1,\ldots, N_\mathrm{body}-1\}\), the following action expressions are introduced,

3.2. State Space

The body link of the snake-like robot are equipped with an accelerometer and a gyroscope (BNO055, Bosch Sensortec GmbH, Germany). The orientation of the gravity acceleration measured by the accelerometer can be used to determine the rotation angle, and the value of the angular velocity measured by the gyroscope can be used to evaluate the moving direction. Therefore, the gravity and angular velocity are included in the state space. The time integration of the target joint angle \(\boldsymbol{\phi}_\mathrm{d}\) shown in Eq. (2) may cause drift. To avoid this, the target joint angle shown in Eq. (1) is also included in the observed values as input to the MLP (Fig. 3). Since the observed gravity acceleration \(\boldsymbol{a}_\mathrm{o}(t)\) and angular velocity \(\boldsymbol{\omega}_\mathrm{o}(t)\) contain noise, the values \(\tilde{\boldsymbol{a}}_\mathrm{o}(t)\) and \(\tilde{\boldsymbol{\omega}}_\mathrm{o}(t)\) obtained by passing through a first-order lag low-pass filter are used as observed values. Here, the time constant of the low-pass filter is \(\Delta T/T_a = 0.2, \ \Delta T/T_\omega = 0.2\). This system is a system in which the action \(\boldsymbol{a}\) corresponds to the target angular velocity \(\dot{\boldsymbol{\phi}}_\mathrm{d}\) of the actuator, and the rolling locomotion is generated by the center of gravity shift due to the deformation of the movable link and the reaction force from the floor. Therefore, it becomes a system in which the change of the action acts with a time delay, and it is necessary to treat it as a Markov decision process with a time delay ³⁵. Therefore, in order to improve the convergence of the reinforcement learning of this system, we introduce a buffered observation ³⁶. In this system, the action \(\boldsymbol{a}\) corresponds to the target angular velocity \(\dot{\boldsymbol {\phi}}_\mathrm{d}\) of the actuator, and the rolling locomotion is generated by the center of gravity shift due to the deformation of the movable link and the reaction force from the floor. Since the rolling lags behind the movement of the links, these observations are buffered and treated as time-series information ³⁶.

The above observations are the same as those used in previous studies for rolling straight ^28,30. Since the purpose of this study is to design a controller capable of not only linear motion but also turning motion, the information indicating the target reaching direction is added to the state space. The overview of the added observations is shown in Fig. 4. The target location \(P_{\mathrm{target}}\) is randomly placed on the circumference of the fixed radius \(R_{\mathrm{target}}\). Since the center of gravity changes continuously due to the movement of the links, the geometric center \(O'\) of the body link is adopted as the physical reference point in the simulation instead of the center of gravity of the robot. The relative vector between the target location and the center of the body link \(O'\) is defined as \(\boldsymbol{r}_{\mathrm{rel}} := P_{\mathrm{target}} - \boldsymbol{r}_{O'}\), and the value obtained by normalizing this relative vector \(\boldsymbol{r}_{\mathrm{rel}}\) by the radius \(R_{\mathrm{target}}\) is added to the state space. To learn the spatial positional relationship, this relative vector \(\boldsymbol{r}_{\mathrm{rel}}\) is transformed into the body coordinate system \(\boldsymbol{r}_{\mathrm{rel,local}}\) using the inverse rotation of the quaternion \(\boldsymbol{q}\) representing the current pose of the robot. The resulting vector, normalized by the radius \(R_{\mathrm{target}}\), is used as the relative bearing vector \(\boldsymbol{o}_5(t_{n})\). This allows the agent to directly observe the relative bearing of the target with respect to its own direction of travel (\(x'\)-axis) rather than the absolute position in the world coordinate system.

Finally, the observations are defined as follows:

3.3. Reward Function

The total reward \(R_n\) at each step is defined as the sum of the step reward \(r_{\mathrm{step},n}\) defined by the product of multiple performance indices, the potential reward \(r_{\mathrm{pot},n}\) based on the change in distance to the target location, and the time penalty \(\rho_{\mathrm{time},n}\) as follows:

3.4. Policy Structure

The policy of the agent is constructed using a MLP. The structure of the network inherits the configuration of a previous study that has an effective observation buffer for systems with time delays. Both the policy network and the value function network consist of fully connected networks with 2 hidden layers of 256 units each. The input observation vector \(\boldsymbol{o}(t_n)\) and the output action \(\boldsymbol{a}(t_n)\) are defined as follows:

4.1. Simulation Setup

To train and evaluate the proposed reinforcement learning algorithm, the physics simulation environment Genesis (v0.2.1) was adopted. Genesis is a physics engine that enables fast computation of multibody dynamics and can simulate the motion model of the snake-like robot described in Section 2. Within the simulator, the interaction between the robot and the floor is calculated based on rigid body dynamics. Considering the stability of the calculations, the time step for physics calculations was set to \(\Delta t = 2.0 \times 10^{-3}\,\mathrm{s}\). On the other hand, the control cycle for policy updates and action decisions by the agent was set to \(\Delta T = 50.0 \times 10^{-3}\,\mathrm{s}\) (\(20\,\mathrm{Hz}\)), which is designed to account for the response characteristics of the actual actuators and the computational load on a SBC. The maximum episode time was set to \(T_{\max} = 30.0\,\mathrm{s}\). However, to improve learning efficiency, if the robot is judged to have fallen, a negative reward is given and the episode is set to end early. The robot is defined as having fallen when the value obtained by passing the acceleration in the \(y'\)-axis (longitudinal direction of the robot) in the body coordinate system through a low-pass filter exceeds 80% of the gravitational acceleration \(g\).

Regarding the observation buffer size used for constructing the state space, \(N_d = 20, N_{\mathrm{skip}} = 2\) was set. The targeted rolling locomotion of the snake-like robot involves a time lag from the driving of the joints to the actual rolling behavior of the robot, so using only the latest observation (\(N_d = 1\)) makes it difficult to learn the causal relationship and degrades movement performance ^28,29,30. In particular, when the buffer size \(N_d\) is small, the lack of past information to compensate for the time delay causes large oscillations in the joint angles and reduces the rolling speed. On the other hand, expanding the buffer size allows for more accurate learning of causal relationships with time delays, and it has been quantitatively demonstrated in previous research ³⁰ that this results in increased cumulative rewards through the generation of smooth movements.

The physical parameters and contact dynamics model used for the analysis are based on the estimation results in a previous study ³⁰ that conducted rolling locomotion experiments using a snake-like robot referenced in this research. Specifically, the friction coefficient \(\mu = 0.10\) and contact resolution time \(\tau_c = 0.01\) s in the contact model were obtained by fitting the time series data of the rotation angle obtained from a free rotation experiment where the snake-like robot was manually rolled on a smooth and flat floor surface with the behavior of the simulation. The reason for narrowing down the environmental characteristics to this single condition is to fix the physical environment that can be physically reproduced and executed as a basis for future verification experiments using the robot. By fixing the environment based on an established experimental setup, the main scope of this paper is to verify the effectiveness of the proposed omnidirectional guidance strategy. While it is difficult to perfectly model the physical contact phenomena and friction characteristics with the ground in the simulation, these identified parameters provide an equivalent contact model that can reproduce the rolling locomotion, and the reproducibility has been experimentally demonstrated in the previous study ³⁰.

4.2. Hyperparameters and Training Settings

The soft actor-critic (SAC) algorithm, an off-policy DRL algorithm, is used for policy learning ^37,38,39. SAC has the characteristic of maximizing the policy’s entropy in addition to maximizing the expected reward, which leads to high exploration efficiency and less susceptibility to local optima. The learning is performed using stable baseline3 (SB3, v2.4.0), and the specific hyperparameters are shown in Table 5 ⁴⁰. The learning rate for both the policy network and the value function network is set to \(3 \times 10^{-4}\), and the size of the replay buffer for storing past experiences is set to \(1.0 \times 10^{6}\). The discount factor for rewards is set to \(\gamma = 0.99\), and a batch of 256 samples is extracted for learning at each update step. By introducing an automatic temperature adjustment parameter for entropy adjustment, it maintains appropriate exploration according to the progress of learning. These parameter values are based on those used in a previous study on the SAC algorithm ³⁹. The training continues until the total reward converges and the success rate of reaching the target location stabilizes sufficiently.

4.3. Initialization

At the beginning of each episode, the \(x'\)-, \(y'\)-coordinates of the center of the robot’s body link \(O'\) are initialized to coincide with the origin \((X, Y) = (0, 0)\) of the world coordinate system. As a reference for the initial orientation, the longitudinal direction of the robot is set to the positive \(X\)-axis direction of the world coordinate system, but the following initial conditions are set to enhance the robustness of learning. The pitch angle of the body is given a random variation based on a uniform distribution in the range of \(\pm 30°\) (\(\pm \pi/6\) rad). This allows the operation to start from different body inclinations for each episode.

5.1. Learning Performance and Convergence

To evaluate the effectiveness of the proposed target-following control method, reinforcement learning was conducted on the physics simulator Genesis. To assess the convergence and stability of learning, five independent learning trials were conducted using different random seeds, and the transition of cumulative rewards is shown in Fig. 5. The solid line in the figure represents the average of the five trials, and the shaded area represents the standard deviation. Examining the learning curves, it can be confirmed that in all trials, the reward stabilizes at a high value (around 270 or above) around \(4.0 \times 10^6\) steps, indicating convergence. Although the target is randomly placed in all directions (azimuth angle \(\theta_{\mathrm{target}} \in [-\pi, \pi)\)) from the early stages of learning, the reward increases smoothly without stagnation, demonstrating that the proposed method has high learning efficiency even for the complex task of omnidirectional target following. This is considered to be because the unit vector of the target direction (from \(\boldsymbol{o}_4\) to \(\boldsymbol{o}_5\)) introduced in the state space can directly and efficiently present the relative directional information of the target location with respect to the agent’s heading direction. Furthermore, the extremely small variance between trials in the later stages of learning indicates that the proposed reward design and network structure have robust learning performance that is not dependent on the initial values.

To evaluate the validity of the convergence value of the total reward around 270, based on the environment specification that the episode ends immediately upon reaching the target, we estimate the theoretical maximum potential of the reward when reaching the target location at the highest speed. As shown later in Fig. 9 (in Section 5.4), the actual arrival time with the acquired policy is about 7 s (equivalent to 350 steps with a control cycle of \(\Delta t = 0.02\,\mathrm{s}\)), at which point the environment resets. With a maximum episode length of 30 s, the remaining time at the point of arrival would be 23 s, resulting in a goal achievement bonus of \(66.0\) and a cumulative time penalty of \(-17.5\) according to Eq. (18). Also, the cumulative potential reward \(r_{\mathrm{pot}}\) when reaching the target location from an initial distance of 3.0 m is calculated as \(50 \times (3.0 - 0) = 150.0\) under the weight coefficient \(w_{\mathrm{pot}} = 50\), as the intermediate term mathematically cancels out. Including the scaling factor \(\alpha = 2\) and considering a good cruising orientation error (\(J_{\mathrm{dir}} = [0.8 \ 1.0]\)) based on the cumulative performance of 20 s in a previous study, which is time-converted to 7 s for this task involving turning, the maximum upper limit estimate of the cumulative step reward \(r_{\mathrm{step}}\) is \(132.3\) (\(49.0 \times 0.9 \times 3.0\)) considering the dynamic characteristics that \(J_{\mathrm{dir}} = 1.0\) cannot be simultaneously satisfied and is limited to about 0.9 in the turning phase where the left turn correction coefficient \(\beta_{\mathrm{left}} = 3.0\) functions due to geometric constraints. Thus, the overall theoretical upper limit range is derived to be approximately 237.7 to 330.8. The fact that the actual learning curve converges around 270, which is a reasonable score range within the upper limit range (247.5 for pure straight cruising), indicates that the acquired control policy has highly learned and utilized the difficult left turn incentive, and has achieved a good balance between rapidity and orientation accuracy in approaching the target location.

5.2. Analysis of Body Link Trajectories

In this section, to evaluate the characteristics of the paths selected by the trained agent for targets in all directions, the trajectories of the body link are analyzed. Fig. 6 shows the body link trajectories when the target is placed at eight target angles (\(\pm \pi/6, \pm \pi/3, \pm 2\pi/3, \pm 5\pi/6\) rad) covering all quadrants centered on the initial position. First, in the forward area where the target exists in the forward direction of the robot (Area A: \(+\pi/6, +\pi/3\) rad and Area D: \(-\pi/6, -\pi/3\) rad), the robot smoothly transitions from the initial forward state to a turning motion, reaching the target location by drawing an arc close to the shortest distance. As the target angle moves away from \(0°\) (approaching the true side), it is confirmed that the turning radius is appropriately adjusted and converges to the target without overshooting.

On the other hand, when the target is placed in the rear area (Area B: \(-2\pi/3, -5\pi/6\) rad and Area C: \(+2\pi/3, +5\pi/6\) rad), the agent selects a characteristic path. In these cases, instead of performing a forward roll and making a large turn, the agent skillfully manipulates the center of gravity shift and the reaction force from the floor to directly orient towards the target direction by rolling backward. Even when the target position changes from the front to the rear, the smoothness of the trajectory is maintained, demonstrating that the proposed method can learn a seamless movement space for all directions without forcing the agent to be aware of discontinuous state changes such as forward or backward. Comparing the trajectories for all eight directions, a high degree of symmetry in the left and right directions can be observed, and there is no bias towards a specific turning direction. This indicates that the policy has overcome the physical characteristic of the robot that tends to turn towards the right direction.

5.3. Motion Analysis Using Snapshots

To clarify the specific control strategies acquired by the trained agent, the behavior of the robot is analyzed using snapshots during the simulation. The turning motion is examined to understand how it is achieved.

5.3.1. Area A (Forward Rolling and Left Turn)

Figure 7 shows a series of snapshots of the turning motion in Area A. This area is a turning phase centered on forward movement, where the robot attempts to turn left while extending the movable link on the right side of the body. By confirming the initial states at \(t=0.30\) s and \(0.63\) s just after the start of the motion, it can be observed that the robot generates turning torque by using specific links as support and achieves rapid heading changes towards the target direction. This tendency can also be confirmed in Fig. 6, where the trajectory extends from the origin towards the target direction even though the robot is initially placed parallel to the \(x\)-axis.

Due to the physical structure of the robot, when the links are extended significantly in the direction of movement for center of gravity shift, a force that deflects towards the right direction naturally acts due to inertia and mass bias. However, in the snapshots at \(t=1.97\) and \(3.97\) s, it was observed that the timing of the leading link contacting the ground is precisely controlled. This control can also be confirmed in Fig. 6, where the trajectory changes sharply towards the target direction at the timing of contact. By converting the ground reaction force obtained at the moment of this contact into a propulsive force towards the left direction, it is considered that the right deflection due to the physical structure is canceled out, allowing for dynamic trajectory correction.

5.3.2. Area C (Backward Rolling and Left Turn)

Next, the analysis results of the left turn motion with backward rolling, which is Area C, are shown in Fig. 8. In this area as well, it can be confirmed that the robot performs heading changes towards the target direction at a very early stage immediately after the start of the motion, achieving rapid direction changes through interaction with the environment. Notably, compared to the snapshots in Area A, it can be observed that even at the same time, the robot has moved to a position closer to the target from the initial position. This is considered to be because Area C is a phase where the robot turns towards the left side where the movable link exists, allowing for more direct and efficient utilization of the mass inertia of the link as a turning moment. Unlike in Area A, the absence of the need for directional adjustment when the movable link contacts the ground is also considered to be one of the factors that enables more efficient turning. In fact, the trajectory in Area C is smoother compared to the trajectory in Area A, and there is no sudden change in direction. Instead, by extending the movable link in the lateral direction, the center of gravity shifts significantly, allowing for heading changes towards the target direction.

5.4. Time-Series Analysis of Joint Angles

The time-series of joint angles \(\phi_{i}\) (\(i=1, \dots, 4\)) during the turning motion for target locations in all eight directions are analyzed to evaluate how the policy manipulates the center of gravity to achieve guidance. Fig. 9 shows the responses of each joint at the time of reaching the target, particularly for the four directions that require significant turning (\(\pm \pi/3, \pm 2\pi/3\) rad) in each area (Areas A–D). Since it is expected that the motion differs for forward or backward rolling and for left or right turns in each quadrant, these four directions were selected for analysis. A characteristic observation is that the time to reach the target in Area D is slower than in the other areas. While it may seem faster to turn right while rolling forward to reach Area D, the robot performs a motion where it rotates more than 90° around the \(Z\)-axis in place in a leftward direction before rolling backward. This motion causes the time to reach the target to be longer.

A common aspect of all motions is that at the start of the motion, the third joint moves with a larger amplitude compared to the other joints. At this time, the robot is performing a direction change near the origin, and as shown in the first snapshot in Figs. 7 and 8, it is supporting itself by contacting the tip of the movable link with the ground. Also, focusing on the time-series of the third joint, it takes time for Areas B and D to enter a periodic state. Since the movable link is good at performing direction changes by receiving a reaction force that kicks the ground in the opposite direction of the robot, it is considered that it is easier to transition from the initial position to the targets in Areas A and C.

Next, comparing the results of Areas A and B, while the previous study only verified forward rolling, in Area B, the target is reached by rolling backward. Although the time to reach the target is almost the same, there is a significant difference in the time-series of joint angles between Area A and Area B. In Area A, the second joint moves significantly, while in Area B, the fourth joint moves significantly. However, this difference in joint movement is not considered to be due to the difference between forward and backward rolling, but rather simply due to the difference in target position. Area D reaches the target by rolling backward, but the waveforms of the joints are similar to Area A. From this, it is considered that the difference between forward and backward rolling is not directly related to the difference in joint movements. Rather, the fact that Area A, which is rolling forward, and Area D, which is rolling backward, have similar time-series of joint angles shows that the same joint movements can lead to reaching the target even with forward or backward rolling.

Finally, regarding the behavior after reaching the target, since the episode ends simultaneously with reaching the target during learning, the behavior after reaching the target is outside the scope of learning. However, it can be confirmed that the joint angles continue to change even after reaching the target. This is because there is no guarantee that the tip of the link is positioned directly under the robot at the time of reaching the target, and rather, it is often the case that the tip of the link is positioned in front or behind the target even after reaching it. Therefore, if the link is suddenly stopped, the robot rotates in a direction away from the target location. To suppress that rotation, it can be confirmed that the link continues to change its joint angles without stopping even after reaching the target location.