2.1. Strain-Sensing Polymer

A strain-sensing polymer is a thin elastic film fabricated by sandwiching a cholesteric liquid crystal (CLC) elastomer film between a pair of elastic polydimethylsiloxane (PDMS) films, and its spectral reflectance properties shift with the magnitude of strain due to the mechano-optical effect of CLC elastomers ¹⁹ and are being investigated as flexible sensor materials ²⁰. The CLC elastomers have a helical molecular orientation, and this helical structure causes wavelength-selective reflection (Bragg reflection). When the pitch of this helical structure is changed by strain, the reflection wavelength shifts. According to ¹⁹, the peak reflectance wavelength of this material is observed at 620 nm without tensile strain, whereas it shifts to a shorter wavelength of 514 nm at 50\(\%\) of tensile strain.

Figure 2 shows how a sheet of the strain-sensing polymer is pulled in a direction parallel to its surface, resulting in a shift of the reflection wavelength toward shorter wavelengths due to lateral strain, and how the entire sheet appears to change color from uniformly reddish to blue in the color image. While these pictures show the change in color caused by transverse strain caused by pulling parallel to the surface of the sheet, the same principle is applied to a load perpendicular to the surface of the sheet, where the sheet is compressed and strained, resulting in a change in the reflected wavelength.

2.2. Structure of the Sensor

Figure 3 shows the structure of the prototype tactile image sensor with strain-sensing polymer sheet. A transparent urethane gel sheet was pasted on a transparent acrylic plate, and a sheet of strain-sensing polymer (thickness 0.275 mm) was placed on top of it. The strain-sensing polymer sheet consists of a CLC elastomer film (0.055 mm in thickness) sandwiched between PDMS films (0.11 mm each in thickness). The surface of the sensor that contacts the object is covered with a black silicone rubber sheet for protection. The strain-sensing polymer was illuminated with white LED light from behind a transparent acrylic plate, and a small color camera (DCT CS-1910-512D-USB) was used to capture images. The camera and the illumination LED are covered with a plastic case to protect it from ambient light. The size of the sensor surface is 15 mm \(\times\) 15 mm.

2.3. Output Images

Figure 4 shows the output color images from the sensor when a hard metal sphere contacts the sensor surface. Before contact (0 N), the color is uniformly reddish. When the sphere is pressed against the sensor surface, the area of contact turns bluish color, and as the force increases (1 N, 5 N, 10 N), the blue color becomes darker and the area of such a bluish region expands. This color change can be detected pixel-by-pixel, allowing the contact area to be detected with high spatial resolution. In order to estimate the pressure distribution, it is necessary to associate the pixel values of a color image composed of RGB values with pressure value.

3.1. Estimating Pressure from Pixel Value in Color Image

It is known that the peak reflection wavelength of the strain-sensing polymer sheet shifts linearly with strain ¹⁹. Therefore, by measuring the reflected light spectrum, the strain can be accurately determined. On the other hand, to realize a small tactile sensor for robotic applications, it is preferable to use a small color camera rather than a spectral measurement instrument. Here, the RGB value obtained from the color image obtained by the small color camera was converted to HSV color representation, and the pressure was estimated using the hue (H) value. The hue value is calculated based on color difference of RGB, so it has larger changes and is more stable. By using the following procedure, the hue value of each pixel is mapped to pressure applied on it.

First, a hard metal sphere (6 mm in diameter) was pressed against the sensor surface, and color images were acquired while varying the vertical force \(F\) from 0 N to approximately 8 N in approximately 0.1 N steps (Fig. 5). In each image, the pixels whose hue changed from the initial no-load condition were extracted, and the corresponding area \(S\) [mm\(^2\)] on the sensor surface was calculated based on the area in pixel on the image. The average pressure in the contact area \(\bar{P}\) is calculated as follows:

3.2. Experimental Results

Several different objects were pressed against the proposed sensor to estimate the pressure distribution on the sensor surface. The RGB color image obtained from the sensor was converted to HSV, and the pressure of each pixel was estimated according to the method described in the previous section. Fig. 7 shows the results of the pressure distribution estimation for three different objects, a metal ball, a discrete transistor with TO-92 package, and an M3 bolt. These objects were pressed against the sensor surface with forces of 7.6 N, 4.8 N, and 1.3 N, respectively.

The pressure distribution estimation results for the metal ball (a) showed a hemispherical pressure distribution as predicted by the Hertz contact stress. In (b), the pitch of three terminals of TO-92 package is 1.27 mm, and these three terminals can be clearly distinguished from the result. Also, the thread pitch of M3 bolt is 0.5 mm, and each thread can be distinguished in (c). Single pixel on the image corresponds to 0.0283 mm \(\times\) 0.0283 mm on the sensor surface, and the pressure value for each pixel ranges up to approximately 0.7 MPa.

3.3. Validation of the Estimated Pressure Distribution

To verify the validity of the estimated pressure distribution, the results of the pressure distribution estimation for the spherical object, shown in Fig. 7(a), were compared with the pressure distribution expected from Hertz’s contact theory. Fig. 8 shows the comparison of the profile of the pressure estimation results (black line) with that expected from Hertz’s contact theory (red line) in 4.2, 6.2, and 7.6 N of the applied force. As the applied force increases, the pressure distribution becomes larger and wider. In all cases, the overall shape was close to the hemispherical pressure distribution as predicted by the Hertz contact stresses. At the edge of the contact area, the estimated pressure is locally high, which is particularly noticeable when the load is 4.2 N. Regarding this, at the edge of the contact area, the strain-sensing polymer sheet may be subjected to not only normal but also shear strain, enhancing the color change of the sheet. CLC elastomers exhibiting a mechanical-optical response have a helical molecular orientation, which causes wavelength-selective reflection. When the pitch of this helical structure changes due to mechanical deformation, the reflected wavelength changes ¹⁹. Based on this, it can be inferred that the shear strain generated at the edge of the contact region contributes to the change in optical properties.

As another verification, the applied force was calculated from the estimated pressure distribution, and compared with true value, which was measured by force gauge (Fig. 5). The estimated force was obtained by integrating the pressure value of pixels whose hue value changed from initial state. Fig. 9 shows the results of comparing the estimated force with the true value measured by the force gauge. The estimated force was generally correct ranging from 0 N to 8 N, and the largest error was approximately 20\(\%\). One possible cause of this error is the error in the estimated pressure value at individual pixels. As shown in Fig. 6, the hue-to-pressure conversion curve used in this study has some errors compared to the pressure calculated from measured force. It is possible that cumulative errors occurred as a result of integrating these to estimate the overall force. Another cause is the influence of the large pressure response shown in Fig. 8, which is presumed to be caused by shear strain appearing at the edge of the contact region. It is possible that the force was overestimated as a result of these responses being included in the integration.

3.4. Spatial Resolution of the Pressure Distribution Sensing

Major advantage of the proposed tactile image sensors is their high spatial resolution in pressure distribution estimation. The size of the sensor surface corresponding to one pixel in the acquired image is 0.0283 mm \(\times\) 0.0283 mm. However, the actual spatial resolution is lower than estimated from this due to the mechanical characteristics of the soft sensor surface material. That is, on an elastomer sensor surface, the deformation at a measurement point is affected by the deformation in its surrounding points. Therefore, we investigated the spatial resolution of the fabricated tactile image sensor through experiments.

The spatial resolution can be evaluated as the minimum distance at which the two points can be distinguished by applying a force to two different points on the sensor surface. Here, we investigated whether the pitch of the threads could be detected correctly in the acquired images when the threads of several bolts with different pitch were pressed against the sensor surface. M3 (pitch 0.5 mm), M2 (pitch 0.4 mm), and M1 (pitch 0.25 mm) bolts were used in the experiment. Fig. 10 shows the pressure distribution for these bolts for different pushing force. For M3 and M2 bolts, the pressure distribution corresponds correctly to the thread pitch when the pressing force is greater than 0.6 N and 0.8 N, respectively, but when the pressing force was lower than that, the thread pitch cannot be identified. As expected from Fig. 6(b), a response corresponding to the pitch of the thread was observed when the pressure exceeded approximately 0.1 MPa. As for the M1 bolt, no response reflecting the pitch of the bolt was observed even if the pushing force increased. The Fourier transform of these responses and examination of the major frequencies revealed that frequencies corresponding to the pitch of the threads were correctly obtained for the M3 and M2 bolts when the pressing force is greater than 0.8 N, but the correct spatial frequencies were not obtained for the M1 bolt. Therefore, the spatial resolution of this tactile image sensor is estimated between 0.25 mm and 0.4 mm at a minimum pressure of 0.1 MPa.

4.1. Algorithm

The high spatial resolution of the proposed tactile image sensor is also useful for accurately estimating the orientation of contacted objects. When an object is grasped by a gripper equipped with a tactile image sensor as shown in Fig. 11, the object’s orientation within the gripper can be accurately estimated, and this information can be used for handling control.

To verify how precisely the orientation can be estimated, we used a thin jumper wire for electronic work as a target object (Fig. 12(a)). Fig. 12(b) shows the output image of the tactile image sensor when the jumper wire was pressed against it with a constant force. Since the black plastic cover of the jumper wire is the thickest part (2.87 mm in diameter), the tactile image sensor primarily responded to that portion. The orientation at this image was defined as 0°, and then the orientation of the jumper wire was varied using a robot arm (UFACTORY xArm 6) to estimate its orientation. For orientation estimation, the template matching method was employed. The image within the white rectangle in Fig. 12(b) was used as the template image, and this template was shifted horizontally and vertically by one pixel and rotated by 0.1° increments while calculating the sum of absolute difference (SAD) value. Both the template and the target images were based on hue images obtained through HSV conversion.

4.2. Experimental Results

Figure 13 shows the experimental results when the jumper wire was placed on the tactile image sensor at orientations of 5°, 10°, and 15°. The template image was rotated, and the minimum SAD value obtained at each angle was plotted. The SAD value reaches its minimum at the region that best matches the template image. Around this region, the rotation step size was set small, while in other ranges a larger step size was used. After calculating SAD values until they became sufficiently larger than the minimum value, a quadratic curve was fitted to the data. The angle at which this quadratic curve reached its minimum was taken as the estimated orientation. As a result, the estimated orientations were 5.35°, 11.40°, and 15.75° for the true orientations of 5°, 10°, and 15°, respectively, corresponding to error rates of 7\(\%\), 14\(\%\), and 5\(\%\).

In the experiment, although the overall load was controlled to remain constant based on the output of the tactile image sensor, slight differences in pressure distribution may have occurred at different orientations. This could be a possible cause of the orientation estimation errors. Also, the estimated orientation had a positive error in all three cases. The cause is unknown, as it is not related to the robot’s positioning accuracy or estimation algorithm. However, it is possible that when the jumper wire was grasped by the robot gripper, the soft surface of the tactile image sensor deformed slightly, causing a slight change in the object’s orientation. On the other hand, in some cases the error was less than 1°, indicating that orientation could be estimated with relatively high accuracy. This capability can be useful for estimating the in-hand position and orientation of grasped objects in robotic hands.