3.1. Fall Risk Alert System ^6,7

This system is designed to run on a smartphone equipped with a light detection and ranging (LiDAR) sensor or a similar technology that generates depth images. The smartphone was attached to the user’s chest, and the recognition space included the ground. The system received several inputs, including depth images obtained from the LiDAR sensor of the smartphone, camera intrinsics, and three-dimensional (3D) poses. The system generates a 3D point cloud from a depth image and camera intrinsics ²³. The generated point cloud is then corrected using the smartphone pose. The point cloud is positioned in a coordinate system, \(\Sigma_{\mathrm{R}}\), such that the height direction is aligned with the gravity direction. A grid map, as shown in Fig. 2, is created by this system using the height values \({}^{\mathrm{R}} z_{\mathrm{R}}\) of this point cloud and the threshold values \(H_{\textrm{chest}}\) and \(L_{\mathrm{w}}\). \({}^{\mathrm{R}}z_{\mathrm{R}}\) represents the height of a point from the ground to the smartphone along the \(\Sigma_{\mathrm{R}}\) coordinate. In addition, the system was designed to alert users while walking. Because the position of the smartphone may change during this activity, the range \(L_{\mathrm{w}}\) is set to reduce misrecognition. Each cell in the grid map has a label classified based on Eq. \(\eqref{eq:1}\), which uses the number of points within a cell, \(n_p\), and the average height within the cell, \(\overline{{}^{\textrm{R}}z_{\textrm{R}}}\). This label includes four types: walkable area \(A_1\), positive obstacle area \(A_2\), negative obstacle area \(A_3\), and occlusion area \(A_4\). These labels are relabeled according to the surrounding state of each cell or the slope angle of the ground set in advance ⁷. The user then sets the slope angle to an acceptable level. In the fall risk estimation described below, negative obstacle area \(A_3\) and occlusion area \(A_4\) are defined as fall risk areas.

The system estimates the fall risk in the walking direction of a user based on the shortest distance \(x_{\mathrm{edge}}\) to the fall risk area and the percentage of the fall risk area on the grid map. \(x_{\textrm{edge}}\) and \(s_{\textrm{risk}}\) were calculated from the generated grid map using Eqs. \(\eqref{eq:1}\) and \(\eqref{eq:2}\), respectively. \(\Delta X\) denotes the grid width in the \(x\)-axis, and \(m_{\textrm{edge}}\) denotes the column index (\(m\)-axis on \(\Sigma_{\mathrm{M}}\)), where the label changes from \(A_1\) to the risk area (\(A_3\) or \(A_4\)). Additionally, \(n_{\textrm{risk}}\) denotes the number of cells with labels \(A_3\) or \(A_4\), and \(N_{\textrm{grid}}\) denotes the total number of cells in the grid map.

The estimated fall risk \(\textit{FR}\) is calculated using Eq. \(\eqref{eq:4}\). \(X_{\min}\) and \(X_{\max}\) denote the lower and upper limits of the grid map, respectively. \(X_{\textrm{th}1}\), \(X_{\textrm{th}2}\), and \(X_{\textrm{th}3}\) (\(X_{\textrm{th}1}>X_{\textrm{th}2}>X_{\textrm{th}3}\)) are the thresholds for \(x_{\textrm{edge}}\), and \(S_{\textrm{th}}\) is the threshold for \(s_{\textrm{risk}}\).

A flowchart of the system is shown in Fig. 3. The system alerts the user only when the estimated fall risk is \(\textit{FR}=1\), 2, or 3 in the walking direction of the user. The system sends alerts via smartphone vibrations. The system can also send alerts via wearable device vibrations that can communicate with a smartphone via Bluetooth. The vibration pattern conveys the changes in the risk level by changing the vibration rhythm according to the estimated risk level, as shown in Fig. 4. The rhythm was set according to haptic icons with rhythm ²⁴. The system is not alert if the risk has the lowest \(\textit{FR}=0\), which allows users to determine the presence or absence of a fall risk based on the presence or absence of vibrations. Users can confirm the detailed conditions of the walking direction if they recognize the vibration rhythm. Additionally, the extra alert level is set to “\(-1\)” for the system, which generates the same alert as \(\textit{FR}= 3\) when the system cannot estimate the risk level. This occurs when the LiDAR sensor on the smartphone does not face the road surface, as determined by the pose of the smartphone.

3.2. Experiment on Station Platform

We conducted experiments using an actual station platform to verify the effectiveness of the proposed system in an actual environment. The verification date was March 23, 2023, and the site was the Hyogo Station platform on the JR-West Wadamisaki Line, as shown in Fig. 5. This experiment was approved by the Institutional Review Board of the Kobe City Medical Center General Hospital. The 10.4 m \(\times\) 3.5 m experimental site on the station platform is shown in Fig. 6. Markers were placed at certain intervals on the ground using tape to identify the positions of the participants during the experiment. The camera was placed at the edge of each experimental site. The camera recorded the participants’ behaviors during the experiment. The participants in this experiment were two right-handed visually impaired individuals (ID 1: a female in her 50s and ID 2: a male in his 70s) who walked alone with a cane on a daily basis and who had no tactile abnormalities or diabetes mellitus. The experiment involved having a participant perform walking-related tasks on a station platform within the experiment site while recording the behavior of the participant and the system. The participants walked through the experimental site according to the two scenarios shown in Fig. 7. In a situation using the system described below, participants paused when they first felt an alert from the system. After pausing, they checked their surroundings and resumed walking. We explained the scenarios and target points described above to the participants before conducting the experiment. For safety reasons, participants wore helmets. During the experiment, a supporter remained on the participant’s side to monitor and prevent them from encountering dangerous situations, such as stepping over the tactile paving. The experimental conditions were combinations of the aforementioned scenarios (two conditions) and the aid usage patterns. The types of aids used were: a cane only (without the system), smartphone \(+\) cane, and smartwatch \(+\) smartphone \(+\) cane. The system alerts are transmitted as vibrations from both smartphones and smartwatches. The smartwatch receives the estimated risk \(\textit{FR}\) as a UTF-8-encoded string via Bluetooth low energy (BLE) from the smartphone. The estimated risk-transmission interval was 0.1 s. The smartwatch then adjusts its vibration pattern based on the string. We used an Apple Watch as the smartwatch. Because the Apple Watch can only generate vibration patterns specific to its operating system, we configured it to produce vibration patterns similar to those corresponding to each level of the estimated risk. We set “start” for \(\textit{FR}=1\), “directionUp” for \(\textit{FR}=2\), and “failure” for \(\textit{FR}=3\) using the vibration patterns from the WatchKit library for Apple Watch ^b. Three trials were performed for each combination. The recorded data included the estimated shortest distance, fall risk area, and fall risk. Data were recorded as CSV files on a smartphone. The parameters for fall risk estimation in an experiment are \(Y_{\textrm{width}}=1.00\) m, \(\Delta X=\Delta Y=0.05\) m, \(L_{\mathrm{w}}=0.10\) m, \(X_{\max}=3.50\) m, \(X_{\textrm{th1}}=3.00\) m, \(X_{\text{th2}}=2.00\) m, \(X_{\textrm{th3}}=X_{\min}=1.50\) m, and \(S_{\textrm{th}}=0.75\). \(H_{\textrm{chest}}\) is adjusted according to participants, with Participant ID 1 set to 1.22 m and Participant ID 2 set to 1.31 m.

A questionnaire was administered to each participant after the experiment. Each participant was asked whether they had stopped based on an alert using a yes/no question to evaluate the effectiveness of the system alerts. We asked each participant to rate their psychological stress on a five-point scale (higher scores indicated lower psychological stress), both when receiving an alert from the system and during the entire task to assess psychological stress during system use. Finally, we administered a system usability scale (SUS) questionnaire to evaluate the system as a welfare device ²⁵. This experiment used the Japanese translation of the SUS ^c as the questionnaire.

5.1. System Behavior on Station Platform

The trends of the system behavior during the experiment were as follows: in Scenario 1, alerts were generated when the participants approached the tactile paving. The fact that system alerts were generated almost simultaneously with the contact with the tactile paving indicates that this system can provide alerts equivalent to those provided by tactile paving on actual station platforms, and when the system is used by visually impaired people. This is also evident from the fact that the area near the platform edge can be labeled correctly in the grid map displayed on the GUI of the application, as shown in Fig. 16. In Scenario 2, the fall risk increased to \(\textit{FR}=1\) only for Participant ID 2. As both participants walked along the tactile paving in the actual walking path, we assumed that an alert was generated only when the participant was walking toward the edge of the platform, as shown in Fig. 17. Therefore, even in situations close to the edge of the platform, the system does not generate an alert unless the walking direction is toward the fall risk area. This is evident from the fact that the grid maps at the start of the tasks are labeled \(A_1\) as shown in Figs. 16 and 17.

We believe that the proposed system can prevent unnecessary alerts.

5.2. Participant Behavior on Station Platform

We observed no clear differences in the walking paths of participants using the system. Because the fall risk estimation is based on the shortest distance in the walking direction, in situations where the participant enters the platform edge at a shallow angle, the system does not generate alerts until approaching the tactile paving. In addition, most questionnaire answers indicated that the participants were unable to stop using an alert from the smartwatch system. We believe that the alert was generated after the cane touched the tactile paving, owing to communication delays. As a countermeasure, the shortest distance threshold can be changed to a longer distance from an alert. However, this method may also cause unnecessary alerts; therefore, care should be taken in the threshold settings. Regarding system usage patterns, according to the questionnaire, the participants answered that they could stop using smartphone alerts. However, they answered that an alert from a smartwatch caused lower psychological stress than an alert from a smartphone alone. This is likely because the smartwatch is constantly in contact with the participant’s skin, generating alerts that are more noticeable than when using a smartphone alone. The inability to stop the smartwatch based on alerts is likely due to communication delays, as previously mentioned. However, the distance between smartphones and smartwatches is small. According to Doe et al., the BLE latency was approximately 5 ms under these conditions ²⁶. Therefore, timing differences due to transmission intervals or other system timing flows may be the cause, instead of poor communication quality. Therefore, addressing these issues is necessary. In actual use, low psychological stress is important when a user uses the system; therefore, we believe that the timing of alerts needs to be adjusted to account for communication delays.

5.3. Effective Situation of Fall Risk Alert System

We confirmed a decrease in the shortest distance and an increase in fall risk near the edge of the platform. Therefore, we consider that an alert similar to tactile paving can provide the same level of alerts on platforms in the same situation as at the experimental site. In addition, because we instructed the participants to perform tasks as similar as possible to their daily activities, we considered that the results of this experiment could be applied to actual use on quiet station platforms.

5.4. Limitation of the Experiment

As a limitation of this experiment, because we conducted it without a human presence, the effects of crowded conditions such as crowds of people could not be considered. In terms of its impact on the system, the sensor recognition space may be obstructed, and countermeasures should be considered to address this issue. In terms of the impact on participants, psychological stress may increase and attention to alerts from the system may weaken owing to the increased presence of elements requiring attention, such as crowds and voices. Because the experiment did not consider situations where trains were stopped, the effect of stopped trains on the estimated results may be uncertain for the system. Participants may also be affected by passenger movement before and after train stops, including station announcements regarding train stops.

5.5. Requirements for Implementation

Based on the above discussion, we consider that the following specific measures are necessary for the practical application of this system: consideration of the timing of the vibration presentation considering communication delays and user perception, countermeasures against crowd occlusion during peak hours, and countermeasures for situations near doors during train stops. We also consider it necessary to explore methods for visually impaired individuals to initiate and terminate the system independently, with the goal of developing a system that can be used by visually impaired individuals without assistance. About the usage pattern of the system, Wong et al. indicated that assistive systems should not provide constant assistance to the visually impaired ²⁷. Not limited to assistive systems, constant support has been indicated as potentially leading to infantilization and self-stigmatization among the visually impaired, as well as loss of independence ²⁸. Therefore, the system should provide alerts in a manner that respects the will of visually impaired users. For instance, when the system detects that a user is approaching a station using the location information of the smartphone, it prompts the user to activate the system. Additionally, we consider that, similar to the use of a cane, users require training for the system to be widely adopted. Tan et al. verified training methods for the visually impaired on the use of smartphone applications and concluded that the optimal content is an individualized approach, graded or scaffolded training, and instructors who are also visually impaired ²⁹. As shown in this example, we should consider it essential to develop training programs during the system implementation phase by incorporating feedback from visually impaired people.