2.1. Model of Optical Reflection Intensity Sensor

We assumed the following model for an optical reflection intensity sensor on the basis of the inverse-square law of light:

2.2. Online Model Acquisition and Distance Measurement

Figure 1 illustrates the three stages of our sensing. First, we acquired the offset parameter \(b\) as the intensity output \(I\) when the object was too far for the intensity sensor to sense (Stage 1). Here, when the sensor is embedded in transparent rubber, as in our previous study ¹⁹, the intensity of the reflected light from bubbles inside the rubber and the rubber surface is included in \(b\). Therefore, it is necessary to use \(I\) while the sensor is emitting light. Subsequently, we acquired the object-dependent parameter \(a\) when the object was sensed by both the intensity and optical ToF sensors (Stage 2). This was conducted by substituting the ToF sensor output \(d_{\rm T}\) [mm] for \(d\) in Eq. \(\eqref{eq:eqnU003Ai-d-eq}\). Finally, we used \(b\) and \(a\) to convert \(I\) to \(d\) when the object was too close for the ToF sensor to detect precisely (Stage 3).

In Stages 2 and 3, the latest \(a\) and Eq. \(\eqref{eq:eqnU003Ai-d-eq}\) are used to calculate the precise distance value \(d_{\rm I}\) [mm] from the intensity output. The equation is explicitly written as follows:

3.1. Compact Package for High Versatility

In our previous study ¹⁹, WrPPS was constructed on the premise of being built into a specific robot hand, resulting in an implementation specialized for that hand. However, in the WrPPS Single Board, by consolidating the minimal configuration of WrPPS (i.e., a combination of one intensity sensor and one ToF sensor) onto a single board, it can be applied wherever this board can be mounted. By minimizing the configuration, a compact board size of \(18~\rm{mm}\times18~\rm{mm}\) was achieved, which is almost equivalent in size to a sensor designed to be mounted on robot fingertips ^a (\(16~\rm{mm}\times18~\rm{mm}\)), providing nearly equivalent mounting possibilities. In addition, three holes with a diameter of 2.2 mm were created to allow secure mounting. The sensors used were, as in our previous study ¹⁹, VCNL4040 (VCNL4040M3OE, Vishay Semiconductors) as the intensity sensor and VL53L0X (STMicroelectronics) as the ToF sensor, and their compact size greatly contributed to the miniaturization of the board.

3.2. Low-Cost Yet Highly Functional Package for High Accessibility

To allow the easy application of the WrPPS Single Board by anyone, we considered it necessary to improve the accessibility of the WrPPS Single Board and began selling the board through the Tokyo Open Source Robotics Kyokai Association ^b. Here, because of the low cost of the component parts, including the sensor elements, and because the assembly was completed through standard PCB assembly, keeping the assembly costs low, the board price was maintained at 7,700 yen, including tax. Comparing this price with other commercially available proximity sensors, it is slightly more expensive than sensors that detect proximity using only an intensity sensor ^a,c at $39.95 (approximately 5,993 yen at 150 yen per dollar) but far less expensive than a sensor based on the high-speed, high-precision proximity-sensing principle ^d at 242,000 yen, making it highly accessible.

Although the former sensors are less expensive, the drawbacks of narrow detection range and strong dependence on the physical properties of the detected object remain because they use only an intensity sensor for proximity sensing, making the WrPPS Single Board meaningful even at a slightly higher price. However, the latest one in the former sensors ^c also includes a pressure sensor, giving it the advantage of being able to readily detect contact pressure. In the future, we aim to introduce a transparent rubber cover as an option for the WrPPS Single Board and implement contact pressure detection using a principle based on measuring rubber deformation ². In addition, the former sensors improved the usability by adopting a standardized communication connector ^e, and the WrPPS Single Board also adopts the same connector.

Although the latter sensor is very expensive, it excels in detection speed and accuracy compared with the WrPPS Single Board. However, the WrPPS Single Board is inexpensive, compact, and capable of long-range detection, making it particularly suitable for applications that require the placement of many sensors on robot surfaces or detection over a wide range.

3.3. ROS-Based Open-Source Software Package for High Adoptability

To improve the accessibility of the WrPPS Single Board from a software perspective, we packaged the software that performed the fusion of the intensity and ToF sensors and released it as open-source software ^f. This software is implemented in ROS, the de facto standard for distributed communication systems within robots, and can consistently handle sensor value acquisition using Arduino, the fusion of the intensity and ToF sensors, and the distribution of distance information. Users can begin the sensor initialization and ROS topic distribution by simply executing the following command:

    roslaunch wrpps_ros \
    wrpps_single_board.launch

This software provides the intensity sensor value \(I\), ToF sensor value \(d_{\rm T}\), distance value \(d_{\rm I}\) derived from the intensity sensor value, and distance value \(d_{\rm c}\) that integrates the ToF sensor value and distance value derived from the intensity sensor value as individual ROS topics. The final output \(d_{\rm c}\) is distributed as a sensor_msgs/Range type, which is the standard ROS distance sensor output format. This output can be obtained as follows:

    $ rostopic echo \
    > /wrpps_single_board\
    > /intensity_model_acquisition\
    > /output/range_combined
    header:
    frame_id: ''wrpps_single_board\
    _intensity_frame''
    field_of_view: 0.44
    min_range: 0.0
    max_range: 2.0
    range: 0.312

Thus, the WrPPS Single Board can be easily integrated into existing ROS-based robot systems and has high adoptability from a software perspective.

4.1. Distance Measurement on Parallel Objects

First, distance measurement experiments were conducted with the sensor board parallel to the measurement target (\(\theta=0°\)). After acquiring \(I\) (intensity sensor output) at a distance \(d=280~\rm{mm}\) from the measurement target as \(b\) (offset parameter), \(d\) was decreased to \(80~\rm{mm}\rightarrow70~\rm{mm}\rightarrow60~\rm{mm}\rightarrow50~\rm{mm}\rightarrow40~\rm{mm}\rightarrow30~\rm{mm}\rightarrow20~\rm{mm}\rightarrow10~\rm{mm}\rightarrow5~\rm{mm}\rightarrow3~\rm{mm}\rightarrow1.95~\rm{mm}\), and 10 output values were recorded at each distance. The minimum distance was 1.95 mm because the connector of the sensor board was 1.95 mm higher than the sensor, causing the connector to contact the target before the sensor. By repeating this procedure of decreasing \(d\) from 80 mm 10 times, 100 (\(10\times10\)) output values were obtained for each distance.

The distance measurement results are shown in Figs. 5–9. For all measurement targets, when the distance was 10 mm or less, the performance of our sensor output \(d_{\rm c}\) exceeded that of the ToF sensor output \(d_{\rm T}\). Fig. 10 describes the intensity sensor output with the offset (\(I-b\)) obtained in this experiment. The intensity value for the pink paper was approximately 10 times as large as the value for the black paper. This difference was too big such that we had difficulty in using these values in the same manner (e.g., using the same threshold for touch detection). On the contrary, our sensor generated similar \(d_{\rm c}\) for the pink and black papers, showing that \(d_{\rm c}\) outperformed the intensity sensor output. For the white PLA sphere, the error of \(d_{\rm c}\) was larger than that for the paper; however, it was still considered small, given that the intensity value was approximately 7 times that for the black paper.

For the A5052 plate, \(d_{\rm T}\) sometimes became an erroneous value (NaN) when attempting to measure at close range. This is likely because the strong specular reflectance of the object caused measurement errors in the ToF sensor. Typically, ToF-sensor measurement errors occur when the measurement target is extremely far from the sensor. Therefore, the current software treats these errors as indications that the measurement target is far away, and following Eq. \(\eqref{eq:eqnU003Acalc-d-c}\), \(d_{\rm c}\) was set to an erroneous value when \(d_{\rm T}\) was an erroneous value. On the other hand, at this time, the distance value \(d_{\rm I}\) converted from the intensity sensor value was normal. If \(d_{\rm I}\) had been output as \(d_{\rm c}\) at this time, although the distance error would have been large, a monotonically decreasing \(d_{\rm c}\) corresponding to a decrease in distance would have been obtained. Therefore, it is necessary to analyze objects with specular reflectance in more detail and improve the switching algorithm between \(d_{\rm T}\) and \(d_{\rm I}\) in the calculation of \(d_{\rm c}\). For the other measurement targets, \(d_{\rm T}\) sometimes had a smaller distance error than \(d_{\rm c}\) at relatively long distances (e.g., \(d=40~\rm{mm}\) in Fig. 6) because the value of \(d_{\rm I}\) rather than \(d_{\rm T}\) was output as \(d_{\rm c}\). Algorithm improvements to address this problem are also required. In addition, for all the measurement targets, \(d_{\rm c}\) decreased less steeply or even increased when \(d\) was reduced to 3 mm or less. This is because, as shown in Fig. 10, the intensity value increased less steeply or even decreased. Therefore, countermeasures such as attaching a cover to the sensor board to limit the approach of objects are required.

4.2. Distance Measurement on Tilted Objects

To investigate the distance measurement performance for objects tilted relative to the sensor, distance measurement experiments were conducted when the angle \(\theta\) of the sensor board relative to the measurement target was not 0°. After setting \(\theta\) to 30° or 60°, the procedure of decreasing \(d\) from 80 mm was repeated ten times, as with the experiments described in the previous subsection. However, because the sensor board has a width rather than being a point and would collide with the measurement target before reaching 1.95 mm, the minimum distance was set to 10 mm when \(\theta=30°\) and 20 mm when \(\theta=60°\). Measurements were performed on four objects shown in Fig. 3, excluding the sphere.

The distance measurement results are shown in Figs. 11 and 12. Even when the sensor was tilted relative to the measurement target, the measured distances did not deviate significantly. For the pink and green papers, \(d_{\rm c}\) decreased as \(\theta\) increased. This is likely because the irradiation range of the ToF sensor spreads rather than remaining at a point. As \(\theta\) increased, the edge of the irradiation range became closer to the sensor than the center of the irradiation range and \(d_{\rm T}\) was considered to become smaller owing to the influence of reflections from there. As \(d_{\rm T}\) became smaller, the acquisition of the object-dependent parameter \(a\) was affected, and \(d_{\rm I}\) also became smaller, which is considered to result in the overall \(d_{\rm c}\) becoming smaller. On the other hand, for the black paper, the change in \(d_{\rm c}\) accompanying the change in \(\theta\) was small, showing that this change became smaller when the reflectance was low. Regarding the A5052 plate, when \(\theta\ne0°\), the distance error and frequency of erroneous values in \(d_{\rm T}\) were reduced compared to when \(\theta=0°\), resulting in an overall improvement in \(d_{\rm c}\). On the other hand, at \(\theta=60°\), \(a\) was never acquired, making it impossible to use \(d_{\rm I}\). This revealed the limitations for objects with specular reflectance.

5.1. Grasping of Compliant Objects

This subsection describes an application example of grasping compliant objects without damaging them using the WrPPS. The details of this subsection can be found in our previous letter paper ¹⁹. In this example, during the grasping motion of a robot hand with WrPPS built into its fingertips, we monitored \(d_{\rm I}\) output from WrPPS. When all values fell below a threshold, a command to stop the fingers was issued, creating a state where the fingertips touched the object gently enough not to crush it, yet firmly enough to lift it.

We conducted five picking-up trials using our precise grasping system against 12 compliant colored boxes. Their compliance came from the thinness of their paper walls (i.e., 0.07 mm). In each trial, we checked if the robot grasped the object gently enough not to squeeze it and firmly enough to lift it up. In addition to the trials using our system, the following cases were tested for comparison:

• picking up only with the intensity sensors against the black and pink boxes, and

• grasping with pressure sensors against the red box.

In the former, our robot hand stopped its fingers when every intensity output with the offset fell below the threshold uniquely defined for each sensor. In the latter, we used the Willow Garage PR2 robot with its pressure sensor fingertips. We utilized the standard PR2 gripper sensor controller with the same thresholds as in the previous study ⁹.

Table 1 shows the results of the task involving the picking up of compliant objects. The deformations by our precise grasping system were smaller than those by the other methods. In addition, our precise grasping was gentle and firm against all box colors.

5.2. Tactile Sensing in a Stuffed Robot

5.2.1. Design of a Tactile Stuffed Robot

For immersive human-robot interaction with a stuffed robot ^21,22, we integrated the WrPPS Single Board to allow a response to being touched by humans. Fig. 13 shows the method for integrating the WrPPS Single Board. By integrating the boards at four locations (i.e., front of the head, back of the head, and both hands), contact can be detected when a person strokes the robot’s head or holds its hands. To mount the sensor boards, the three holes described in Section 3.1 were used. Secure mounting of the sensor boards was achieved by threading strings through these holes and sewing them onto the outer skin. Because the sensor board was compact, the inhibition of the flexibility of the outer skin caused by sewing it was suppressed. In addition, because the sensor board was thin, no protrusions were formed on the stuffed robot surface, thereby suppressing the sense of discomfort when stroking. These properties helped maintain the stuffed toy’s natural appearance and feel during human-robot interaction.

5.2.2. Stroke Detection Experiment

We conducted an experiment in which the robot moved in response to a person stroking its head. In the experiment, the robot responded to four of five stroking motions (Fig. 14), confirming that even soft contact when stroking a stuffed toy could be detected. Even in the only motion where contact was not detected, there was a sensor response. Therefore, this can be addressed by improving the contact detection method.

5.3. Slip Detection During Walking

5.3.2. Walking Hand Prototype

Figure 15 shows a 4-finger hand for walking with the index, middle, ring, and little fingers with the WrPPS Single Board attached to the fingertip of the middle finger. The surface of the sensor board was covered with a flexible UV-curing resin (LED Curing Resin “Star Drop” Gummy, PADICO) to allow frictional contact with the ground. Because the sensor board was compact, it was possible to attach it to the small fingertip and execute the same walking motion as before the sensor board was attached.

5.3.3. Slip-Detection Experiments

We conducted slip-detection experiments during the walking motion (Fig. 16). The sensor value waveforms differed significantly between the cases with and without slip, confirming that they were distinguishable. In addition, walking motions were performed for four cycles each, and the difference in waveforms between cycles was small, confirming that stable discrimination was possible.

5.4. Agricultural Plant Sensing

5.4.2. Agricultural Plant Sensing Experiments

Therefore, it is necessary for proximity sensors to detect plant branches (i.e., thin objects). In proximity sensor research, there are examples of pen slippage detection using visual sensing ¹³ and electrical wire detection using ToF sensing ²⁵. However, these were limited to detecting objects within a close range of less than 40 mm, and the detection performance for more distant objects was unclear. Therefore, we investigated whether the intensity and ToF sensors could detect plant branches using the WrPPS Single Board. The experimental setup is shown in Fig. 17. The detection target was an actual grapevine stick (diameter 10 mm). The stick was placed directly in front of the sensor board, and while varying the distance from the sensor board, the intensity and ToF sensor values were measured 50 times at each distance. The minimum distance was 10 mm and the maximum was 100 mm, and measurements were taken in 10 mm increments. The experimental results are shown in Fig. 18. It was observed that both the intensity and ToF sensors could detect the stick. However, the ToF sensor output distances greater than the actual distance, indicating reduced accuracy.