Top > JRM > Goal-Conditioned Self-Supervised Learning from Play Aimed at H ...

single-rb.php

JRM Vol.38 No.5 pp. 1402-1412
(2026)
doi: 10.20965/jrm.2026.p1402

Paper:

Views over last 60 days: 10

Goal-Conditioned Self-Supervised Learning from Play Aimed at Human–Robot Collaboration

Shun Hiramatsu and Shingo Murata ORCID Icon

Keio University
3-14-1 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa 223-8522, Japan

Received:
September 12, 2025
Accepted:
July 3, 2026
Published:
October 20, 2026
Creative Commons License
Keywords:
robot learning, cognitive robotics, human–robot collaboration, predictive coding, learning from play
Abstract

Robots capable of accurately understanding situations based on environmental information and collaborating effectively with humans in a variety of tasks are in high demand. Deep-learning-based approaches can be effective for realizing such collaborative robots; however, collecting data and achieving high generalization performance are challenges. To address this issue, we utilize “play data” collected by teleoperating a robot when performing collaborative tasks with a human partner. This study proposes a play-supervised latent goal for human–robot collaboration (play-HRC) framework that learns from unlabeled play data for human–robot collaborative scenarios without explicit goal observation. Unlike play-supervised latent motor plan that generates actions conditioned on an externally specified goal observation, play-HRC must infer a latent goal from an initial state. At the beginning of the task execution, play-HRC infers an achievable goal from its initial state. During execution, it generates actions and predicts future environmental states while adaptively optimizing the latent goal by minimizing prediction errors. To evaluate the performance of play-HRC, we conducted tests in a real-world environment using a task in which a robot and human collaboratively manipulate objects. The results highlighted the ability of the framework to infer achievable goals, generate actions, and dynamically adjust goals in response to human behavior.

Human-robot collaboration from play

Human-robot collaboration from play

Close full text
Cite this article as:
S. Hiramatsu and S. Murata, “Goal-Conditioned Self-Supervised Learning from Play Aimed at Human–Robot Collaboration,” J. Robot. Mechatron., Vol.38 No.5, pp. 1402-1412, 2026.
Data files:

1. Introduction

In the near future, collaborative robots are expected to work alongside humans in shared environments 1. By sharing tasks within a shared workspace, humans and robots are expected to increase the work efficiency. If a robot is only required to perform pre­determined tasks, rule-based control may be sufficient. However, there are situations where various tasks must be performed within a workspace. In such cases, it is necessary for humans and robots to share task goals, and for the robot’s actions to be planned accordingly 2,3.

The methods for sharing task goals between humans and robots can be divided into two categories: goals explicitly informed to robots in advance by humans using language or other means, and goals where the robot itself infers the implicit goals from environmental information. Although generating robot behavior based on language- or image-defined goals has been actively studied in recent years 4,5,6,7, the present study focuses on the latter scenario because in the real world, there are many situations in which people work by implicitly sharing goals with each other. Therefore, instead of explicitly conveying goals to a robot, this study proposes a framework in which both humans and robots infer achievable task goals based on the current environment, match these goals as they proceed with the task, and ultimately achieve shared goals. This study aims to construct a framework that enables robots to generate actions by inferring achievable goals from their current environmental states and adapting them based on human behavior.

Reinforcement learning (RL) 8,9 and imitation learning (IL) 10,11,12 involving deep learning methods are effective means of achieving this goal. However, they also have limitations. RL leads to a high generalization performance through exploration, but its learning efficiency is low. Moreover, contact with humans during the exploration stage can be dangerous, and designing rewards in a real-world environment is difficult. In contrast, IL involving demonstration data in places where humans operate robots are efficient but lead to low generalization performance and requires labeling of data, which is a labor-intensive task.

Deep learning-based approaches have demonstrated impressive capabilities; however, real-world human–robot collaboration (HRC) scenarios still present two key challenges: (i) designing dense, safety-aware reward functions for RL, and (ii) acquiring sufficiently diverse, labeled demonstration data for IL. One potential solution to these challenges involves “play data” 13—unstructured data collected when robots are operated by humans driven by curiosity or intrinsic motivation. For example, if a box is placed to the right of a ball, several reachable next states are possible—such as placing the ball in the box, to the right of the box, on top of the box, or underneath it. The operator can freely choose between these outcomes when teleoperating a robot. Even after selecting a specific goal, the operator retains discretion over how to achieve it by relying on their knowledge and intuition regarding the environment. In other words, play data represent intentional and goal-directed behavior and not random behavior. Because the temporal sequence of the proprioceptive and visual data is recorded, manual labeling is not required, and data reflecting the operator’s knowledge of object manipulation can be obtained. The present work builds on this concept by leveraging unlabeled play data to retain the expressive power of deep networks while avoiding reward engineering and manual annotation.

Lynch et al. 13 introduced play data for robot learning and proposed a play-supervised latent motor plan (play-LMP) framework. Their approach involved two deep networks: one that inferred a latent plan from the entire subsequence of proprioception and image data, and the other that inferred it using only the initial and final parts of the subsequence. They trained these networks using unlabeled play data and performed various tasks successfully. However, it is difficult to apply this framework directly to HRC because it does not support dynamic goal modification during task execution.

For a robot to collaborate with humans, it must infer human intentions from environmental information and optimize its own task goals based on this inference. One possible approach for implementing this mechanism is predictive coding (or predictive processing) 14,15, in which the brain continuously generates top-down predictions of sensory input and minimizes bottom-up prediction errors. This idea, influential in areas such as neuroscience and cognitive science, has also been adopted in the field of cognitive robotics 16,17. Higher-level internal representations generate predictions concerning the dynamics of lower-level processes extending to the sensory-motor level. Prediction errors at the sensory-motor level are propagated upward through hierarchical structures, utilizing the observed outcomes to adjust the internal states and minimize errors. This enables the robot to adapt to unexpected situational changes in the real-world environment 18,19.

Motivated by play-LMP and predictive coding, we propose play-HRC, a framework for HRC without an explicit goal observation that differs from play-LMP in three aspects:

  1. ・

    Goal inference timing. At the beginning of the task execution, play-HRC infers a distribution over the reachable latent goals from the initial visual state. The goal inference network is applied only during this initial step.

  2. ・

    Goal update capability. While play-LMP generates actions conditioned on an externally specified goal observation, play-HRC infers a latent goal from the initial state and couples it with a predictive coding loop. Prediction errors between the predicted and observed states drive gradient-based updates of a latent goal, thereby enabling dynamic goal adaptation.

  3. ・

    Human–robot collaboration. The predictive coding mechanism is inherently sensitive to exogenous disturbances, particularly human intervention, in the workspace. When such events occur, the resulting prediction errors trigger a rapid goal realignment, allowing seamless adaptation without explicit human input.

In practice, play-HRC employs an end-to-end deep neural network that (i) infers achievable goals from the current environment, (ii) generates low-level actions conditioned on these goals, and (iii) continuously updates the goals through prediction-error minimization (PEM). We validate the proposed framework using a real-world object manipulation task, in which a robot must dynamically adjust its behavior in response to human intervention. Our results demonstrate the practical benefits of online, time-varying goal optimization in collaborative settings.

To evaluate the effectiveness of play-HRC, we conducted an experiment in a real-world setting of a robotic task involving object manipulation and assessed the goal-inference capabilities of the proposed method.

2. Related Work

2.1. Learning from Demonstrations

IL has enabled robots to perform a variety of object manipulation tasks 20,10,21,22,23. A common strategy for improving generalization is to train high-capacity models on large, diverse datasets 5,6,24. For example, RT-X 24 was trained on a dataset containing 527 skills (160,266 tasks) collected from 22 different robots in collaboration with 21 research institutions. However, such large datasets require the manual preparation of robot demonstrations for each task. By contrast, the present study leverages play data 13 to reduce the burden of preparing task-specific demonstrations. Pretrained generalist robot policies and vision-language-action (VLA) models, such as RT-2 and Octo, have been used to transfer large-scale visual, language, and robotic experiences to robot control and adaptation 6,23. More closely related to online RL, RL Token 25 uses a pretrained VLA model to reduce the number of online interactions required. However, these approaches rely on language or goal-image conditioning, task rewards, or additional real-world interactions, whereas play-HRC learns from unlabeled play data and evaluates latent-goal inference and revision without an explicit goal observation.

2.2. Learning from Play

Several studies have explored multi-task learning using play data, which exhibits a broad applicability 13,26,27,4. The concept of play data has broad applicability. In addition to play-LMP 13, as described in the previous section, prior studies have combined offline RL with hierarchical structures to successfully perform long-horizon tasks 26, and have applied domain transfer techniques to human play data for learning robotic object manipulation 4. Conventional demonstrations are generally collected for predefined tasks, whereas play data consist of continuous, unsegmented interactions that are not labeled by the task. Therefore, the two data sources are complementary. Representation learning from visual data is also relevant to learning from play. The R3M 28 learns reusable visual representations from diverse human videos for downstream robot manipulation. This approach complements play-HRC, which trains its visual encoder using robot play data. Although these studies achieved high performance in diverse tasks, they did not focus on human collaboration or on fluid task execution with an understanding of human intentions. Therefore, this study proposes a method that combines goal and intention inference with task learning based on play data.

2.3. Inferring Human Intention and Prediction-Error Minimization

As humans are increasingly integrated into robot workspaces, the inference of human intentions has emerged as a critical area of interest 1. Recently, several deep learning-based methods have been proposed 29,30,31. Recurrent neural networks (RNNs) and graph convolutional networks are commonly used and have been shown to classify complex human behaviors using cues such as gaze and body movements 32.

When intention inference is primarily achieved through RNNs, incorporating predictive coding enables feedback from the prediction results into the robot’s action planning. Collaborative work between robots and humans can be realized by inferring human intentions and generating actions based on them. Previous studies have successfully optimized the prediction of both environmental changes and human intentions, using these predictions to generate appropriate actions 33,34. These studies demonstrated that neural networks can capture temporal and stochastic structures and adapt to changing goals by updating their internal states to minimize prediction errors. In our study, we also introduce predictive coding as an algorithm for optimizing work goals.

3. Method

3.1. Preliminaries

The proposed method, play-HRC, is a framework designed for human–robot collaboration, well-suited for learning from play data. Let \(\mathcal{D}_{\mathrm{play}}\) denote the play dataset, consisting of a sequence of observation–action pairs: \(\mathcal{D}_{\mathrm{play}} = \{(O_{1}, a_{1}), \ldots, (O_{T}, a_{T})\}\).

Here, \(O_{t} = \{o_{t,1}, \ldots, o_{t,N}\}\) denotes the image observations from \(N\) cameras at time step \(t\), and \(a_t\) denotes the robot’s action (joint angles) at the same time. Following Lynch et al. 13, learning is performed by sampling a subsequence beginning at time step \(t\) with length \(\kappa\), where \(\kappa_{\mathrm{min}} \le \kappa \le \kappa_{\mathrm{max}}\).

In our framework, we define a “goal” as the final state to be reached from the current state.

3.2. Play-HRC

An overview of the proposed framework, play-HRC, is illustrated in Fig. 1.

figure

Fig. 1. Overview of play-HRC. During training, the goal-recognition network encodes a sampled subsequence and its distribution is matched to that of the goal-inference network. At the beginning of a test episode, the goal-inference network is applied once to initialize \(z_{\mathrm{infer}}\). The goal-optimization module then updates the latent goal online using the state-prediction error, and the optimized goal \(z'_{\mathrm{infer}}\) is supplied to the policy network. Dashed boxes indicate training-only components.

Play-HRC differs from play-LMP 13, which generates actions conditioned on an externally specified goal observation. It enables action generation conditioned on a goal inferred from the initial state, using goal-conditioned self-supervised learning on play data. Moreover, play-HRC leverages prediction errors between predicted and actual environmental states to dynamically update inferred goals in response to human actions.

Play-HRC comprises the following elements. First, the vision network extracts the image feature state \(s_{t}\) from the image observation \(O_{t}\) and outputs the reconstruction \(\hat{O}_{t}\). The goal-inference network extracts the latent representation of the goal (latent goal) \(z_{\mathrm{infer}}\) from the image feature state \(s_t\), and the goal-recognition network is used for its training. The policy network outputs the predictions of the next action \(\hat{a}_{t+1}\) and the image feature state \(\hat{s}_{t+1}\) based on the current action \(a_t\), the image feature state \(s_t\), and the inferred latent goal \(z_{\mathrm{infer}}\). During training, one latent goal \(z_{\mathrm{recog}}\) is sampled for each subsequence and remains fixed within that subsequence. At test time, the goal-inference network is applied once to the initial image feature state \(s_0\) to initialize \(z_{\mathrm{infer}}\). The goal-inference network is not applied again during the episode; instead, the latent goal is updated online through prediction-error minimization 35.

3.2.1. Vision Network for Encoding and Decoding Visual Observations

We define the vision network to encode and decode image observations. The vision network consists of an encoder \(E = \{E_{1}, \ldots, E_{N}\}\) that extracts the image feature state \(s_{t}\) from \(O_{t}\) and a decoder \(D = \{D_{1}, \ldots, D_{N}\}\) that reconstructs \(\hat{O}_{t}\) from \(s_{t}\). Each encoder \(E_i\) uses an EfficientNet-B0 backbone 36, initialized with ImageNet weights and fine-tuned during training. Each decoder \(D_i\) is a three-layer transposed convolutional network. Each encoder–decoder pair is defined per camera input and maps high-dimensional images to low-dimensional features, then reconstructs them from these features.

\begin{align} s_{t} &= \operatorname{concat} \left( \left[{E_{1}\bigl(o_{t,1}\bigr)}, \ldots, {E_{N}\bigl(o_{t,N}\bigr)} \right]\right) , \label{eq:eq1} \tag{1} \end{align}
\begin{align} \hat{O}_{t} &= \left\{{D_{1}\left(E_{1}\bigl(o_{t,1}\bigr)\right)}, \ldots, {D_{N}\left(E_{N}\bigl(o_{t,N}\bigr)\right)} \right\} . \label{eq:eq2} \tag{2} \end{align}
The reconstruction error during training is expressed by the following equation.
\begin{equation} L_{\mathrm{recon}} = \frac{1}{\kappa} \sum_{\tau=t}^{t+\kappa-1} \operatorname{MSE}\left(\hat{O}_{\tau}, O_{\tau}\right) . \label{eq:eq3} \tag{3} \end{equation}

3.2.2. Goal-Inference Network for Inferring Goals from a Current State

The goal-inference network \(p_{\mathrm{infer}}\) is a stochastic feedforward network with two fully connected layers, from which reachable latent goals \(z_\mathrm{infer}\) are sampled conditioned on the image feature state \(s_{t}\). This network maps \(s_{t}\) to the parameters \(\mu_{\mathrm{infer}}\) and \(\sigma_{\mathrm{infer}}\) of the latent goal distribution. Similar to a variational autoencoder 37, this network serves as a prior, assumed to follow a normal distribution:

\begin{equation} p_{\mathrm{infer}}\bigl(z|s_{t}\bigr) = \mathcal{N}\bigl(\mu_{\mathrm{infer}}, \sigma_{\mathrm{infer}}\bigr). \label{eq:eq4} \tag{4} \end{equation}

3.2.3. Goal-Recognition Network for Sequence Encoding

The goal-recognition network is a stochastic transformer with two encoder layers 38,27, from which latent goals \(z_{\mathrm{recog}}\) are sampled conditioned on the entire sequence. A transformer encoder is adopted to capture long-range temporal dependencies that span hundreds of steps in a play sequence—critical for goal recognition. Self-attention is effective at capturing such dependencies while being permutation-invariant with respect to sequence length. It maps the entire input sequence to the parameters \(\mu_{\mathrm{recog}}\) and \(\sigma_{\mathrm{recog}}\) of the latent goal distribution. Similar to the goal-inference network, this network serving as an approximate posterior distribution is assumed to follow a normal distribution, given by the following equation:

\begin{equation} q_{\mathrm{recog}}\bigl(z|a_{t:t+\kappa}, s_{t:t+\kappa}\bigr) = \mathcal{N}\bigl(\mu_{\mathrm{recog}},\sigma_{\mathrm{recog}}\bigr). \label{eq:eq5} \tag{5} \end{equation}

The latent goal \(z_{\mathrm{recog}}\) is sampled using the reparameterization trick during training and is input to the policy network. Note that this network is not used at the test time.

3.2.4. Prior Matching

During training, the divergence between the approximate posterior \(q_{\mathrm{recog}}\) and the prior \(p_{\mathrm{infer}}\) in latent space is minimized. This is done by minimizing the Kullback–Leibler divergence \(L_{\mathrm{KL}}\), defined as

\begin{equation} L_{\mathrm{KL}} = D_{\mathrm{KL}} \left(q_{\mathrm{recog}}\bigl(z|a_{t:t+\kappa}, s_{t:t+\kappa}\bigr) || p_{\mathrm{infer}}\bigl(z|s_{t}\bigr)\right). \label{eq:eq6} \tag{6} \end{equation}

Here, \(D_{\mathrm{KL}}(q || p)\) represents the Kullback–Leibler divergence between two distributions \(q\) and \(p\). This ensures that the latent goal \(z_{\mathrm{infer}}\) reachable from the current state \(s_{t}\) can be sampled.

3.2.5. Policy Network for Predicting Future States

The policy network \(\pi\) is a three-layer RNN with gated recurrent units (GRUs) 39, which predicts the next action \(\hat{a}_{t+1}\) and image feature state \(\hat{s}_{t+1}\). Combining action and state prediction in a single network ensures internal consistency and facilitates shared feature extraction. While self-attention is powerful for sequence summarization, real-time control demands millisecond-level inference. GRUs offer a favorable speed-accuracy trade-off and reduced memory usage, making them well-suited for closed-loop control on embedded systems. These predictions are conditioned on the current action \(a_t\), image feature state \(s_t\), and latent goal \(z\). During training, the latent goal is \(z_{\mathrm{recog}}\); during testing, it is \(z_{\mathrm{infer}}\).

The loss function during training is computed as the negative log-likelihood for each time step in the sequence, as follows:

\begin{equation} L_{\mathrm{policy}} = - \frac{1}{\kappa} \sum_{\tau=t}^{t+\kappa-1} \log{\pi} \bigl(a_{\tau+1}, s_{\tau+1} | a_{\tau}, s_{\tau}, z\bigr) . \label{eq:eq7} \tag{7} \end{equation}

3.2.6. Model Training

Play-HRC is trained end-to-end, with each network optimized to minimize the sum of the loss terms described above.

\begin{equation} L = L_{\mathrm{recon}} + L_{\mathrm{policy}} + \beta L_{\mathrm{KL}} \tag{8} \end{equation}
Here, following Higgins et al. 40, we introduce a weight \(\beta\) to control the contribution of \(L_{\mathrm{KL}}\) to the overall loss. Setting \({\beta < 1}\) helps prevent a common issue in variational autoencoder training, where a powerful decoder ignores the latent variable—resulting in “posterior collapse.” In this study, \(\beta\) was empirically set to 0.25. We also incorporate KL balancing to stabilize training 41,42.

We use AdamW 43 as the optimizer and cosine annealing 44 for learning rate scheduling. The subsequence length was sampled from \(\kappa_{\mathrm{min}}=90\) to \(\kappa_{\mathrm{max}}=125\).

3.3. Goal Optimization

At the beginning of testing, the goal-inference network is applied once to \(s_0\), and the latent goal \(z_{\mathrm{infer}}\) is sampled from \(p_{\mathrm{infer}}(z|s_{0})\). At each subsequent time step, the policy network takes \(a_t\), \(s_t\), and the current optimized latent goal \(z'_{\mathrm{infer}}\) as inputs and predicts \(\hat{a}_{t+1}\) and \(\hat{s}_{t+1}\). The prediction error is computed over a recent time window from time step \({t-w+1}\) to \(t\), as follows:

\begin{equation} L_{\mathrm{pred}} = \frac{1}{w} \sum_{\tau=t-w+1}^{t} \operatorname{MSE}\left(s_{\tau+1}, \hat{s}_{\tau+1}\right) . \label{eq:eq9} \tag{9} \end{equation}
Accumulating prediction errors over recent steps allows the framework to refine the goal based on both current and past states, thereby improving action generation accuracy.

The gradient of the prediction error with respect to the current latent goal is computed through the policy network, and the goal is updated in the opposite direction of the gradient as follows:

\begin{equation} z'_{\mathrm{infer}} = z_{\mathrm{infer}} - \alpha \frac{\partial L_{\mathrm{pred}}}{\partial z_{\mathrm{infer}}}. \tag{10} \end{equation}
Here, \(\alpha\) is a hyperparameter that determines the update rate. The update is repeated \(n\) times within each time step, and the resulting \(z'_{\mathrm{infer}}\) is supplied to the policy network. In the experiments, the time-window length, update rate, and number of updates were set to \({w=10}\), \({\alpha=0.01}\), and \({n=30}\), respectively. Without PEM, the latent goal \(z_{\mathrm{infer}}\) inferred from the initial state remains fixed throughout the episode.
figure

Fig. 2. Experimental environment. Left: experimental setup. The experimental space is a table with one green cup, two pink cups, and one blue container operated by a human. The human and the robot face each other across the table. The cups and container are manipulated by the robot and the human, respectively. Right: Four states included in the play data. \(S_1\) represents the state where the green cup is on the left from the robot’s direct view, \(S_2\) represents the state where the green cup is in the middle, \(S_3\) represents the state where the green cup is on the right, and \(S_4\) represents the state where all cups are stored in the blue container.

4. Experiments

4.1. Experimental Setup

The experimental setup is shown in Fig. 2 (left). The space is set up with one green cup, two pink cups, and one blue container, and the robot and the human are facing each other across the space. In this space, the robot and the human collaborate to manipulate the cups. Additionally, images are captured using two cameras for an overhead view and a side view.

We used the WidowX 250 Robot Arm 6DOF manufactured by Trossen Robotics. The robot has seven degrees of freedom in total: six joints and one gripper. However, since the fifth joint remains static throughout the tasks in this study, it was excluded from the training data and replaced with a fixed value during testing. For image acquisition, we used the Intel RealSense Depth Camera D435. Although the camera provides RGB images at a resolution of \({480 \times 640 \times 3}\), the images were resized to \({128 \times 128 \times 3}\) to accelerate training.

The operator can teleoperate the robot based on their own curiosity, but for the sake of simplifying the learning process, there are three restrictions:

  1. ・

    The cups are manipulated by the robot and the container is manipulated by the human.

  2. ・

    When swapping the positions of two cups, the operation must start from the green cup, and the space in the center back must be used.

  3. ・

    If the human places the container, the robot must manipulate cups to place them in the container.

These constraints were introduced because this study remains exploratory. Our objective is to evaluate the fundamental effectiveness of the proposed play-HRC framework before extending it to large-scale, unconstrained collaborative scenarios.

4.2. Dataset and Task Design

The dataset consists of approximately 2 h of play data, which are used as training data, and 90 labeled data, which are used as evaluation data. The tasks performed during the collection of play data can be described as follows. First, consider four states: \(S_1\), \(S_2\), \(S_3\), and \(S_4\), as shown in Fig. 2 (right). Here, \(S_1\) represents the state where the green cup is on the left from the robot’s direct view, \(S_2\) represents the state where the green cup is in the middle, \(S_3\) represents the state where the green cup is on the right, and \(S_4\) represents the state where all cups are stored in the blue container. In this case, the goal from \(S_i\) to \(S_j\) is represented as \(G_{i \to j}\).

Considering the three states excluding \(S_4\) as initial states, there are a total of nine goals, which are differentiated based on the presence or absence of human intervention. \(G_{1 \to 4}\), \(G_{2 \to 4}\), and \(G_{3 \to 4}\) are classified as tasks with human intervention, where the human places the container in the space and the robot manipulates the grasped cups to place them in the container. The remaining goals are tasks without human intervention, where the robot grasps and manipulates each cup and swaps their positions.

4.3. Experimental Criteria

We conducted experiments to investigate the effectiveness of the proposed method in the experimental environment and with the task design described above. We did not directly compare play-HRC with play-LMP because play-LMP generates actions conditioned on an externally specified goal observation, whereas the present setting provides no explicit goal observation to the robot. Therefore, to isolate the contribution of dynamic goal revision under the same no-goal-input setting, we compared play-HRC with and without PEM. In both conditions, the goal-inference network inferred a latent goal from the initial state; with PEM, the latent goal was optimized online, whereas without PEM, it remained unchanged.

Therefore, to evaluate the usefulness of the proposed method, we investigated the following:

  1. ・

    Whether the goal-inference network can infer reachable goals from the initial state at task onset.

  2. ・

    Whether the goal can be modified by minimizing the prediction error.

  3. ・

    Whether it is possible to generate appropriate actions based on the inferred goal.

5. Results and Discussion

5.1. Inferring Goals from Current States

figure

Fig. 3. The distribution of latent goals \(z_{\mathrm{recog}}\) and \(z_{\mathrm{infer}}\) sampled from the goal-recognition network and the goal-inference network in the latent space. For each initial state, the distribution of latent representations \(z_{\mathrm{recog}}\) (filled points) and \(z_{\mathrm{infer}}\) (unfilled points) in the latent space is presented. The color of the filled points is based on the final state. The points labeled as 1a and 1b, 2a and 2b, and 3a and 3b represent the sampled latent goals \(z_{\mathrm{infer}}\) from \(S_1\), \(S_2\), and \(S_3\), respectively, during the test in a real-world environment, with two trials conducted for each state. The corresponding generated actions are shown in Fig. 4.

figure

Fig. 4. Examples of generated actions inferred from two possible goals for each initial state. For example, in trial 1a, the robot moved the green cup from the left to the center and the state transitioned from \(S_1\) to \(S_2\). Note that the corresponding latent goal \(z_{\mathrm{infer}}\) that represents the transition from \(S_1\) to \(S_2\) is shown in Fig. 3. Similarly, in the other trials, the goal was inferred from the initial state and actions were generated, indicating that the inferred goal was achieved.

To investigate whether the goal-inference network can infer reachable goals from the current state, we repeatedly sampled latent goals from the goal-recognition network/goal-inference network using various states as inputs. We then observed the results of these samples to evaluate the network’s performance. Fig. 3 presents the distribution of the latent goals \(z_{\mathrm{recog}}\) and \(z_{\mathrm{infer}}\) by the goal-recognition and goal-inference networks in the latent space, respectively. First, \(z_{\mathrm{recog}}\), which is represented by filled points, is clustered according to the final state, and the distribution of \(z_{\mathrm{infer}}\), which is represented by unfilled points, is included in the distribution of \(z_{\mathrm{recog}}\). We explain this result, using the example of initial state \(S_1\). From \(S_1\), the reachable goal states include \(S_2\), \(S_3\), and \(S_4\). The latent goals \(z_{\mathrm{recog}}\) extracted from sequences transitioning from \(S_1\) to each goal state (\(S_2\), \(S_3\), and \(S_4\)) are separated by the differences in goal states. Additionally, the distribution of the inferred latent goal \(z_{\mathrm{infer}}\) from the initial state \(S_1\) is included in the distribution of \(z_{\mathrm{recog}}\). This indicates that the goal-inference network can infer reachable goals from the current state.

Next, we focus on the results in the real-world environment. The points 1a, 1b, 2a, 2b, 3a, and 3b in Fig. 3 represent the inferred goals \(z_{\mathrm{infer}}\) from each state in the real-world environment, and the results of action generation at that time are shown in Fig. 4. For example, in trial 1a, the robot moved the green cup from the left to the center and the state transitioned from \(S_1\) to \(S_2\). Fig. 3 shows that at this time, the robot infers the achievement of \(S_2\) as the goal. Similarly, in the other trials, the goal is inferred from the initial state and the actions are generated, indicating that the inferred goal is achieved. From these results, it is demonstrated that play-HRC can infer reachable goals from the current state and generate appropriate actions in the real-world environment.

figure

Fig. 5. The difference in action generation when human intervention occurs (top: with prediction-error minimization, bottom: without prediction-error minimization). In both scenarios, at time step \({t=25}\), the human has placed the container on the table, and the robot needs to manipulate cups to place them in the container. With prediction-error minimization, the robot successfully manipulates each cup to the container and achieves the task. In contrast, without prediction-error minimization, collisions occurred between the container and the robot arm, resulting in failure.

5.2. Optimizing Goals Under Human Intervention

We evaluate the optimization performance of the goals when a person intervenes in the task by comparing the behavior with and without prediction-error minimization. As shown in Fig. 5, when a person intervened in the task, there was a difference in the success or failure of action generation depending on the presence or absence of prediction-error minimization. When prediction-error minimization was employed, play-HRC was able to manipulate each cup and place it into the container placed by the person in the middle of the table, whereas when prediction-error minimization was not employed, the arm collided with the container after manipulating the first cup, resulting in failure. This failure likely resulted from the need to generate a trajectory that does not collide with the container, which differs from the case where there is no container in the space.

The results of goal inference and the temporal change of prediction error are shown in Figs. 6 and 7. When prediction-error minimization is employed, \(G_{1 \to 3}\) is inferred from the initial state and subsequently updated to \(G_{1 \to 4}\) (\(z'_{\mathrm{infer}}\) in Fig. 6 and the upper-left panel in Fig. 7). The prediction error increases rapidly after the person places the container on the table at time step \({t=25}\) and subsequently decreases (bottom panel in Fig. 7 (left)). When prediction-error minimization is not employed, \(G_{1 \to 3}\) is inferred from the initial state and remains unchanged thereafter (\(z_{\mathrm{infer}}\) in Fig. 6 and the upper-right panel in Fig. 7). The prediction error also increases after the person places the container on the table and subsequently decreases (bottom panel in Fig. 7 (right)). Even when no prediction-error minimization is employed, the prediction error decreases and the robot correctly performs the task until the halfway point (placing the first cup into the container). This is because the policy network receives not only the latent goal but also image features and actions at the current time, so it is often possible to make some correct predictions. However, the goal has not changed despite human intervention, so a mismatch between the latent goal and the current state is considered to have occurred, resulting in failure. Therefore, it is essential to optimize the goal so that it corresponds to the change in state following human intervention.

figure

Fig. 6. The difference in the latent goal when human intervention occurs. With prediction-error minimization, the optimized goal \(z'_{\mathrm{infer}}\) transitions from \(G_{1 \to 3}\) to \(G_{1 \to 4}\). Without prediction-error minimization, the latent goal \(z_{\mathrm{infer}}\) inferred from the initial state remains associated with \(G_{1 \to 3}\). Note that the other latent goals, derived from the goal-recognition network, are identical to those for the initial state \(S_1\) in Fig. 3.

figure

Fig. 7. Dynamics of the latent goal and prediction error when human intervention occurs at time step \({t=25}\). The upper panels show the PC1 and PC2 scores of the latent goal obtained using the same PCA transformation as in Fig. 6; the lower panels show the prediction error. Left: with PEM. Right: without PEM.

5.3. Task Success Based on Inferred Goals

Table 1 shows the success rates of the proposed method. In the experimental setting of this study, when a person intervened, the goal was unambiguously determined, but when no intervention occurred, there were multiple possible goals. Additionally, the proposed method, play-HRC, infers reachable goals from the initial state at task onset and generates appropriate actions without pre-defined goals. Therefore, in cases where no intervention occurred, we consider it a “success” if one of the reachable goals from that initial state (e.g., the state \(S_1\) can reach the state \(S_2\) or \(S_3\)) was achieved within a specified time. Under these conditions, we conducted 30 trials for each case.

Table 1. Comparison of task-success rates with and without prediction-error minimization (PEM).
Without human With human
With PEM 60.0% 53.3%
Without PEM 63.3% 36.7%

Comparing the success rates of the proposed method based on whether or not prediction-error minimization was employed, there was no difference in the success rates for tasks without human intervention, whereas for tasks with human intervention, the success rate increased with prediction-error minimization. These results suggest the effectiveness of the proposed method in inferring goals from the initial state at task onset and generating appropriate actions as well as the usefulness of prediction-error minimization. More specifically, PEM had little effect without human intervention: 18 of 30 trials succeeded with PEM and 19 of 30 succeeded without PEM. Under human intervention, however, the number of successful trials increased from 11 of 30 without PEM to 16 of 30 with PEM. In the failure example in Fig. 5, the condition without PEM continued behavior associated with the latent goal inferred from the initial state after the container was introduced, which could produce a trajectory inconsistent with the changed environment and lead to collision. Because trial-level failure causes were not annotated systematically, this interpretation is limited to the observable behavior and aggregate success rates.

6. Conclusion

In this study, we proposed a framework for enabling collaborative tasks between robots and humans, using unlabeled “play data” instead of individually collected demonstrations for each task. Motivated by play-LMP, we constructed a network that infers reachable goals from the initial state without explicit goal observation and generates appropriate actions. In addition, we introduced prediction-error minimization to enable adaptation to goal changes. By introducing learning with play data and dynamic goal optimization with prediction-error minimization, the framework successfully enabled interactive collaborative work with humans. Experimental results for tasks involving collaboration between humans and robots demonstrated the effectiveness of the proposed framework.

We also realized collaborative tasks with humans by learning using play data. However, to simplify the experiments, several constraints were imposed and the number of cups in the test environment was reduced. Owing to these constraints and simplifications, the patterns of movements achievable in the environment were also limited, and we were unable to fully utilize the richness of the play data, which reflects the operator’s curiosity and knowledge of object manipulation. Therefore, in future work, we aim to minimize the restrictions on robot operation in environments with multiple objects and achieve various collaborative tasks with humans present in it.

Acknowledgments

This work was supported by the Japan Science and Technology Agency (PRESTO Grant Number JPMJPR22C9), JSPS KAKENHI Grant Numbers JP24K03012 and JP26H02515, and the Kayamori Foundation of Informational Science Advancement.

References
  1. [1] D. Mukherjee, K. Gupta, L. H. Chang, and H. Najjaran, “A survey of robot learning strategies for human-robot collaboration in industrial settings,” Robot. Comput.-Integr. Manuf., Vol.73, Article No.102231, 2022. https://doi.org/10.1016/j.rcim.2021.102231
  2. [2] A. Bauer, D. Wollherr, and M. Buss, “Human–robot collaboration: A survey,” Int. J. Humanoid Robot., Vol.5, No.1, pp. 47-66, 2008. https://doi.org/10.1142/S0219843608001303
  3. [3] L. Contreras, J. Savage, S. Ortuño-Chanelo, M. Negrete, and H. Okada, “Complex task planning for general-purpose service robots,” J. Robot. Mechatron., Vol.37, No.3, pp. 594-603, 2025. https://doi.org/10.20965/jrm.2025.p0594
  4. [4] C. Wang et al., “MimicPlay: Long-horizon imitation learning by watching human play,” Proc. 7th Conf. Robot Learn., pp. 201-221, 2023.
  5. [5] A. Brohan et al., “RT-1: Robotics transformer for real-world control at scale,” Robotics: Science and Systems XIX, 2023. https://doi.org/10.15607/RSS.2023.XIX.025
  6. [6] B. Zitkovich et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” Proc. 7th Conf. Robot Learn., pp. 2165-2183, 2023.
  7. [7] T. Motoda, T. Kitamura, R. Hanai, and Y. Domae, “SuctionPrompt: Visual-assisted robotic picking with a suction cup using vision-language models and facile hardware design,” J. Robot. Mechatron., Vol.37, No.2, pp. 374-386, 2025. https://doi.org/10.20965/jrm.2025.p0374
  8. [8] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” Int. J. Robot. Res., Vol.32, No.11, pp. 1238-1274, 2013. https://doi.org/10.1177/0278364913495721
  9. [9] J. Ibarz et al., “How to train your robot with deep reinforcement learning: Lessons we have learned,” Int. J. Robot. Res., Vol.40, Nos.4-5, pp. 698-721, 2021. https://doi.org/10.1177/0278364920987859
  10. [10] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, “Learning and generalization of motor skills by learning from demonstration,” 2009 IEEE Int. Conf. Robot. Autom., pp. 763-768, 2009. https://doi.org/10.1109/ROBOT.2009.5152385
  11. [11] T. Zhang et al., “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” 2018 IEEE Int. Conf. Robot. Autom., pp. 5628-5635, 2018. https://doi.org/10.1109/ICRA.2018.8461249
  12. [12] T. Yamashita, H. Suzuki, and R. Tasaki, “Skill implemented motion planning for precision assembly task of part mating gears,” J. Robot. Mechatron., Vol.37, No.2, pp. 434-443, 2025. https://doi.org/10.20965/jrm.2025.p0434
  13. [13] C. Lynch et al., “Learning latent plans from play,” Proc. 3rd Annu. Conf. Robot Learn., pp. 1113-1132, 2019.
  14. [14] R. P. N. Rao and D. H. Ballard, “Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects,” Nat. Neurosci., Vol.2, No.1, pp. 79-87, 1999. https://doi.org/10.1038/4580
  15. [15] A. Clark, “Whatever next? Predictive brains, situated agents, and the future of cognitive science,” Behav. Brain Sci., Vol.36, No.3, pp. 181-204, 2013. https://doi.org/10.1017/S0140525X12000477
  16. [16] A. Ciria, G. Schillaci, G. Pezzulo, V. V. Hafner, and B. Lara, “Predictive processing in cognitive robotics: A review,” Neural Comput., Vol.33, No.5, pp. 1402-1432, 2021. https://doi.org/10.1162/neco_a_01383
  17. [17] T. Taniguchi et al., “World models and predictive coding for cognitive and developmental robotics: Frontiers and challenges,” Adv. Robot., Vol.37, No.13, pp. 780-806, 2023. https://doi.org/10.1080/01691864.2023.2225232
  18. [18] J. Tani, “Learning to generate articulated behavior through the bottom-up and the top-down interaction processes,” Neural Netw., Vol.16, No.1, pp. 11-23, 2003. https://doi.org/10.1016/S0893-6080(02)00214-9
  19. [19] S. Murata et al., “Learning to perceive the world as probabilistic or deterministic via interaction with others: A neuro-robotics experiment,” IEEE Trans. Neural Netw. Learn. Syst., Vol.28, No.4, pp. 830-848, 2017. https://doi.org/10.1109/TNNLS.2015.2492140
  20. [20] P. Englert and M. Toussaint, “Learning manipulation skills from a single demonstration,” Int. J. Robot. Res., Vol.37, No.1, pp. 137-154, 2018. https://doi.org/10.1177/0278364917743795
  21. [21] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” Proc. 1st Conf. Robot Learn., pp. 357-368, 2017.
  22. [22] N. M. M. Shafiullah et al., “On bringing robots home,” 2023. https://doi.org/10.48550/arXiv.2311.16098
  23. [23] D. Ghosh et al., “Octo: An open-source generalist robot policy,” Robotics: Science and Systems XX, 2024. https://doi.org/10.15607/rss.2024.xx.090
  24. [24] A. O’Neill et al., “Open X-Embodiment: Robotic learning datasets and RT-X models: Open X-Embodiment Collaboration,” 2024 IEEE Int. Conf. Robot. Autom., pp. 6892-6903, 2024. https://doi.org/10.1109/ICRA57147.2024.10611477
  25. [25] C. Xu et al., “RL Token: Bootstrapping online RL with vision-language-action models,” arXiv:2604.23073, 2026. https://doi.org/10.48550/arXiv.2604.23073
  26. [26] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard, “Latent plans for task-agnostic offline reinforcement learning,” Proc. 6th Conf. Robot Learn., pp. 1838-1849, 2023.
  27. [27] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto, “From play to policy: Conditional behavior generation from uncurated robot data,” arXiv:2210.10047, 2022. https://doi.org/10.48550/arXiv.2210.10047
  28. [28] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3M: A universal visual representation for robot manipulation,” arXiv:2203.12601, 2022. https://doi.org/10.48550/arXiv.2203.12601
  29. [29] S. Li, L. Zhang, and X. Diao, “Deep-learning-based human intention prediction using RGB images and optical flow,” J. Intell. Robot. Syst., Vol.97, No.1, pp. 95-107, 2020. https://doi.org/10.1007/s10846-019-01049-3
  30. [30] L. Geng, W. Yang, Y. Jiao, S. Zeng, and X. Chen, “A multilayer human motion prediction perceptron by aggregating repetitive motion,” Mach. Vis. Appl., Vol.34, No.6, Article No.98, 2023. https://doi.org/10.1007/s00138-023-01447-6
  31. [31] J. Tang, J. Wang, and J.-F. Hu, “Predicting human poses via recurrent attention network,” Vis. Intell., Vol.1, No.1, Article No.18, 2023. https://doi.org/10.1007/s44267-023-00020-z
  32. [32] P. Schydlo, M. Rakovic, L. Jamone, and J. Santos-Victor, “Anticipation in human-robot cooperation: A recurrent neural network approach for multiple action sequences prediction,” 2018 IEEE Int. Conf. Robot. Autom., pp. 5909-5914, 2018. https://doi.org/10.1109/ICRA.2018.8460924
  33. [33] S. Murata et al., “Achieving human–robot collaboration with dynamic goal inference by gradient descent,” Proc. 26th Int. Conf. Neural Inf. Process., Part 2, pp. 579-590, 2019. https://doi.org/10.1007/978-3-030-36711-4_49
  34. [34] A. Ahmadi and J. Tani, “A novel predictive-coding-inspired variational RNN model for online prediction and recognition,” Neural Comput., Vol.31, No.11, pp. 2025-2074, 2019. https://doi.org/10.1162/neco_a_01228
  35. [35] K. Friston, “The free-energy principle: A unified brain theory?,” Nat. Rev. Neurosci., Vol.11, No.2, pp. 127-138, 2010. https://doi.org/10.1038/nrn2787
  36. [36] M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks.” Proc. 36th Int. Conf. Mach. Learn., pp. 6105-6114, 2019.
  37. [37] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” Proc. 2nd Int. Conf. Learn. Represent., 2014. https://doi.org/10.48550/arXiv.1312.6114
  38. [38] A. Vaswani et al., “Attention is all you need,” Proc. 31st Int. Conf. Neural Inf. Process. Syst., pp. 6000-6010, 2017.
  39. [39] K. Cho et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” Proc. 2014 Conf. Empir. Methods Nat. Lang. Process., pp. 1724-1734, 2014. https://doi.org/10.3115/v1/D14-1179
  40. [40] I. Higgins et al., “beta-VAE: Learning basic visual concepts with a constrained variational framework,” 5th Int. Conf. Learn. Represent., pp. 60-81, 2017.
  41. [41] O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning over unstructured data,” IEEE Robot. Autom. Lett., Vol.7, No.4, pp. 11205-11212, 2022. https://doi.org/10.1109/LRA.2022.3196123
  42. [42] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering Atari with discrete world models,” arXiv:2010.02193. 2020. https://doi.org/10.48550/arXiv.2010.02193
  43. [43] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 7th Int. Conf. Learn. Represent., 2019.
  44. [44] D. Zhang, D. Dai, Y. He, F. S. Bao, and B. Xie, “RLScheduler: An automated HPC batch job scheduler using reinforcement learning,” Int. Conf. High Perform. Comput. Netw. Storage Anal., 2020. https://doi.org/10.1109/SC41405.2020.00035

Creative Commons License  This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Oct. 19, 2026