1. Introduction

Recently, owing to the rapid advancement of artificial intelligence (AI) and robotics technologies, the range of services that robots can perform in daily human life has significantly expanded ¹. For example, robots that provide services traditionally performed by humans, such as caregiving robots in home environments, guide robots at tourist sites, and customer service robots in stores and restaurants, are rapidly becoming integrated into everyday life ^2,3,4. These service robots are drawing considerable attention as promising solutions to labor shortages in aging societies ^5,6,7.

However, despite the widespread adoption of these service robots, users often report difficulties in perceiving human warmth or empathy in the services provided by robots ^8,9. This issue is considered not merely a matter of functionality, but rather a consequence of the fact that current service robots tend to engage in mechanical and one-directional interactions, often failing to fully comprehend the emotional context of human users ¹⁰.

Tomasello explained the origins of human language in terms of the development of cooperative sociality and shared intentionality, arguing that the essence of language lies in the capacity for mutual understanding and coordination toward shared goals ^11,12. In particular, emotional attunement enables interlocutors to infer the affective state of speakers and thereby achieve a more accurate understanding of their internal condition. Accordingly, emotional attunement can be regarded as a key component of not only human–human interactions but also realizing natural communication in human–robot and human–system interactions. In this study, emotional attunement is not treated as direct emotion recognition. Instead, we define emotional attunement through the inference of user attentional intention derived from gaze behavior. Because gaze reflects attentional focus and cognitive engagement, predicting gaze trajectories enables the system to estimate the attentional focus of a user in real time. This attentional intention is used as a practical proxy for emotionally attuned interactions, allowing the system to adapt its responses based on the anticipated user focus rather than explicit emotional labels. By adopting this formulation, the present study emphasizes an engineering-oriented approach to nonverbal interactions, focusing on attention-aware support rather than affective state estimation.

In human–human interactions, emotional attunement is facilitated by the continuous interpretation of multiple nonverbal cues ¹³. Individuals indirectly infer the internal state of another person by observing subtle changes in facial expressions, shifts in gaze direction, variations in posture, and dynamic gestures that accompany speech or contextual behavior. These nonverbal signals provide rich information on affective intent, engagement levels, and underlying psychological conditions, enabling interlocutors to construct a more precise understanding of the emotional state of the other person. This multimodal information plays a crucial role in maintaining coordinated interactions and sustaining mutual understanding. Building on the view that emotional attunement in human communication is achieved through the continuous interpretation of nonverbal cues, this study positions gaze behavior as a particularly informative channel for extending these mechanisms to service-oriented robotic systems, as illustrated in Fig. 1.

Because gaze reflects the attentional focus, cognitive processing, and affective engagement of a user, analyzing gaze patterns offers a promising means to infer the internal state of the user in contexts where direct verbal expressions may be limited or ambiguous. Therefore, we developed a range of robotic systems that leverage nonverbal information, particularly gaze-based cues, to facilitate more adaptive, context-aware, and emotionally attuned interactions with users ^14,15,16. Building on these prior developments, the present study introduces a gaze-based analytic approach designed to enhance the ability of robot systems to interpret the internal states of users for emotionally attuned interactions.

The remainder of this paper is organized as follows. Section 2 describes the proposed systems used in this study. Section 3 describes the structure of the machine learning system. Section 4 presents an analysis of the experimental results. Section 5 discusses directions for future work.

2. Overall System Configuration

In this study, we employ a smart home system as an interactive platform. This system is designed for domestic environments and supports remote operations. We use HoloLens 2 ¹⁷, a head-mounted display capable of mixed reality (MR) interactions, as the input device for interacting with the smart home system. By incorporating the MR technology, the system can utilize digital twin representations, allowing users in remote locations to visually access real-time information on the operation of their smart home devices.

To ensure applicability across diverse residential environments, the proposed system controls home appliances using attachable modules that can be retrofitted into general household settings. The target appliances include curtains and lighting that are frequently used in daily life, and the system additionally provides access to the internal state of a refrigerator ⁷. By monitoring the operational states of these appliances, we can make a simple estimation of the lifestyle patterns of residents within the environment.

The overall system flow is illustrated in Fig. 2. When input device (1) HoloLens 2 is activated, (2) information about the actual home appliances is retrieved from the database, and the virtual appliances displayed in the HoloLens 2 environment are updated to match their physical counterparts. When the user issues an operation command through HoloLens 2, the command is transmitted to the database, and (3) a robot or ESP32-S3 accesses the database to operate the corresponding physical appliances. Both HoloLens 2 and ESP32-S3 periodically communicate with the database, enabling real-time remote interactions ¹⁸.

In this system, the robot functions as an information mediator for users in the physical environment, as illustrated in Fig. 3, by notifying them of appliance activation events and providing additional appliance-related information ¹⁹. The virtual environment displayed in the HoloLens 2 is constructed based on the physical environment in which the appliances are installed, as shown in Fig. 4. The basic behaviors of the virtual robot partner are illustrated in Fig. 5. To enable remote control within HoloLens 2, the system measures both the hand movements and gaze behavior of the users. The users can operate all the appliances using a combination of hand gestures and gaze input. In particular, gaze information is used not only for appliance manipulation but also for enabling the robot to provide informational support.

Gaze information is obtained using two infrared sensors integrated into HoloLens 2. As shown in Fig. 6, the gaze data are collected and analyzed, and the point of gaze contact is indicated by a red dot. After completing the calibration procedure, the system accurately measures the gaze of each user. The gaze point is computed by identifying the position at which the two independently captured gaze rays from the left and right eyes are closest to each other. To determine the gaze point of the user within the virtual environment, the gaze coordinates obtained from the local head coordinate system of the head-mounted display are transformed into the world coordinate system.

3. Machine Learning Model

In this study, a machine learning model is designed to predict future gaze points to infer user attentional intention rather than directly estimating emotional states. By analyzing time-series gaze data, the system estimates the forthcoming focus of attention of a user that is used as an operational proxy for emotional attunement in human–system interactions. This prediction enables the robot partner to anticipate the user interests and provide adaptive, context-aware information support.

To achieve this, we employ a machine learning framework that combines a long short-term memory network with a mixture density network (LSTM–MDN) to model the temporal dynamics and uncertainty of gaze behavior.

The overall architecture of the model is illustrated in Fig. 7. The model aims to predict future gaze point \(\mathbf{g}_{K+1} \in \mathbb{R}^3\) based on a sequence of past observations. This study uses a dataset consisting of 9,233 gaze sequences in which each sequence represents a set of gaze data points sampled at 0.3 s intervals. The dataset is divided into training, validation, and testing sets at ratios of 70%, 15%, and 15%, respectively. Let \(\mathbf{g}_t = (x_t, y_t, z_t)^\top\) denote the gaze coordinates at timestep \(t \in \{1, \dots, K\}\), where \(\top\) denotes the transpose operator and \(K=10\) is the sequence length. To capture the dynamics of gaze behavior, we compute velocity \(\mathbf{v}_t = (v_{x,t}, v_{y,t}, v_{z,t})^\top\) and acceleration \(\mathbf{a}_t = (a_{x,t}, a_{y,t}, a_{z,t})^\top\) using finite differences with a sampling interval of \(s = 0.3\) s as follows:

To extract the temporal dependencies from input sequence \(\mathbf{U}\), we employ an LSTM network. Recurrent computations within the LSTM unit at each timestep \(t\) are governed by the following equations.

To account for the multi-modalnature and inherent uncertainty of gaze movements, the MDN maps hidden state \(\mathbf{h}_K\) to the parameters of a Gaussian mixture model (GMM) with \(M=5\) components. The MDN estimates mixing coefficients \(\pi_m\), means \(\boldsymbol{\mu}_m = (\mu_{m,x}, \mu_{m,y}, \mu_{m,z})^\top\), and standard deviations \(\boldsymbol{\sigma}_m = (\sigma_{m,x}, \sigma_{m,y}, \sigma_{m,z})^\top\) for each component \(m \in \{1, \dots, M\}\) using MDN-specific parameters \(\mathbf{W}^{(M)}\) and \(\mathbf{b}^{(M)}\):

4. Experimental Results

In the present study, the experimental evaluation intentionally focused exclusively on gaze information to isolate attentional intention as the primary nonverbal cue. Although emotional communication in human interactions is inherently multimodal, incorporating facial expressions, speech, and body gestures, the purpose of this experiment was to examine the feasibility of attention-aware support based solely on gaze behavior. Multimodal affective analyses and comparative evaluations with and without gaze information are beyond the scope of the current study and are reserved for future studies.

4.1. Experimental Overview

Based on this design choice, we conducted an experiment to perform real-time gaze prediction using the constructed learning model. Real-time gaze coordinates were transmitted from HoloLens 2 to the server that processed the input data to predict the results. In this system, gaze measurement data were sent to the server every 0.3 s, and the system yielded predictions of gaze coordinates 0.3 s into the future. The 0.3-s interval was selected by considering the average duration of human eye blinks and was thus adopted as the gaze measurement period in the system. Note that the measurement samples in which the gaze coordinates were located extremely far from the virtual environment displayed in the virtual space were excluded from use.

To obtain the results as accurately as possible, the experiment was conducted in a quiet environment without any objects obstructing the field of view of the participant. During the experiment, each of the three appliances targeted for operation—refrigerator, curtain, and lighting—was used at least once. Because the system performed prediction every 0.3 s, the duration of each experiment was set to at least 30 s.

As shown in Fig. 8, the experimental procedure consisted of (1) a tutorial explaining the method of using the system followed by (2) the experiment itself. Finally, (3) the experimental results were computed and the prediction accuracy was evaluated. Fig. 9 shows the experimental setup in action. Specifically, (a) presents an external view of the user during the experiment, and (b) displays the system interface from the user perspective.

4.2. Gaze Prediction Performance

The experiment was conducted with 14 participants in their twenties. For each experiment, the deviation between the measured and predicted gaze coordinates was calculated. The averaged results are shown in Fig. 10; the horizontal axis represents the error distance between the predicted and measured coordinates, and the vertical axis represents the cumulative ratio.

To assess the overall predictive performance, we reported the 70th percentile error that represented the error distance within which 70% of all predictions fell. This percentile-based metric was adopted to robustly characterize the typical prediction accuracy under the multimodal distribution modeled by the MDN. A percentile-based evaluation is particularly suitable for gaze prediction in which error distributions are often skewed and multimodal.

As shown in Fig. 10, the 70th percentile error in a three-dimensional space was 0.29 m. To illustrate the scale of this distance, the size of the virtual environment constructed in the system is shown in Figs. 11 and 12. Fig. 11 shows the virtual environment from the front in which the room width and height are 0.35 m and 0.24 m, respectively. Fig. 12 shows the environment from above in which the room depth is 0.56 m. These results indicated that an error of 0.29 m in a three-dimensional space corresponded to 18.5% of the maximum possible error.

Next, we considered the error in two-dimensional planes. The average errors in each plane computed from the experimental results are shown in Fig. 10. As shown in Fig. 13, on the \(X\)–\(Y\) plane, the 70th percentile error was 0.09 m, and the corresponding values on the \(Y\)–\(Z\) and \(X\)–\(Z\) planes were 0.26 m and 0.29 m, respectively. These results indicated that the errors in the depth direction (\(Z\)-axis) were relatively large, whereas the errors in the horizontal \(X\)- and vertical \(Y\)-directions were smaller.

One factor contributing to the larger error in the depth direction was the perceptual error in recognizing the distances within the virtual space ^20,21. As revealed in the observational evaluation, participants often perceived the buttons to be located closer than their actual positions when attempting to press them. This suggested that the human perceptual error in the \(Z\)-direction was relatively large, which resulted in the observed discrepancy between the predicted and measured values.

Despite the larger prediction error observed along the depth, that is \(Z\)-axis, the proposed system remained capable of supporting intention-level interactions because appliance discrimination in a smart home environment primarily relied on spatial separation in the \(X\)–\(Y\) plane. In practice, the predicted gaze information is used to estimate the general user focus rather than precise depth positioning, allowing the system to tolerate \(Z\)-axis uncertainty while still providing effective context-aware assistance.

This study predicted gaze coordinates 0.3 s into the future. However, for more adaptive attention-aware interaction, the prediction horizon must be extended beyond 0.3 s to longer time intervals. Therefore, extending the prediction horizon (for example, to 0.9 s and 1.5 s) is considered important for future personalization and will be addressed in future studies.

4.3. Application Example of Gaze-Based Adaptive Information Support

To illustrate using predicted gaze information in practical interactions, this subsection presents an application example of gaze-based adaptive information support using a robotic partner. Within the system, a robot provided information and proposed services to a user. This robot introduced home appliances located in the direction of the gaze of the user. By tuning this function according to the results of gaze prediction, the system could offer services that were better tailored to each user.

As a basic form of information provision, as presented in Table 1, each home appliance provided various types of information based on the current time. In addition to these gaze prediction results, we believed that more personalized information could be provided by referring to the routine-based behavioral schedule of a user in a real environment. For example, as illustrated in Fig. 14, when the predicted gaze position was near the refrigerator and the current time was 7:00 AM, the schedule indicated breakfast time. In this case, the robot was expected to provide information regarding the ingredients in the refrigerator that were typically used for breakfast, thereby realizing a service adapted to the situation of the user.

5. Conclusion and Future Works

In this study, we proposed a gaze-based user analysis system integrated into an MR smart home environment to support attentional intention-aware human–system interactions. Rather than directly estimating emotional states, the system inferred the attentional intentions of users from their gaze behavior as an operational proxy for emotional attunement, thereby enabling adaptive and context-aware information support.

The proposed system enabled remote operation of home appliances through a virtual environment while continuously measuring gaze information to predict future gaze coordinates 0.3 s ahead using an LSTM–MDN model. Experiments conducted with 14 participants demonstrated that 70% of predictions fell within an error of 0.29 m in three-dimensional space, with notably higher accuracy on the \(X\)–\(Y\) plane. These results indicated that gaze-based attentional intention prediction could be achieved with sufficient accuracy for interactive assistance; however, depth-related uncertainty remains a limitation that requires further consideration. By sharing the predicted gaze information with a robotic partner, the system could anticipate user focus and provide proactive situation-aware support. This approach demonstrated the feasibility of leveraging gaze prediction for attention-aware interactions in MR-based smart environments.

Future work will extend the prediction horizon beyond 0.3 s, incorporate additional nonverbal modalities, and evaluate the system with broader user populations, including elderly participants, to improve generalizability. Furthermore, adaptive service strategies that integrate predicted gaze with contextual information such as time and user routines will be explored to enhance proactive assistance, interaction efficiency, and context-aware system responses in real-world smart home scenarios. Furthermore, adaptive service strategies that integrate predicted gaze with contextual information such as time and user routines will be explored to enhance proactive assistance, interaction efficiency, and context-aware system responses in real-world smart home environments.