1. Introduction

The recent decline in working-age population in many developed countries is severe owing to the decrease in population. The reduction in the number of workers and the aging of the labor force pose significant challenges especially in the agricultural sector. Although IoT technology is applied across many fields and is expected to resolve problems associated with the declining birthrate and aging population, many issues remain to be addressed in the agricultural sector. The industrial robots used in the manufacturing sector handle materials with defined specifications, or objects with high rigidity. However, agricultural products are diverse in shape, and soft vegetables and fruits are easily damaged, thus requiring precise control of the gripping force ¹. To enable autonomous agricultural robots to handle delicate crops, such as strawberries and tomatoes, without causing damage, integrated sensors must satisfy certain specific functional requirements. A force resolution of less than 0.1 N is required to detect the initial contact and to precisely adjust the gripping force ². Furthermore, a rapid sensor response is essential for achieving real-time feedback control to prevent slippage during the movement of the robotic hand ^3,4. Humans can hold soft objects without crushing them by integrating and processing information obtained through vision and touch. Robots require sensors that mimic both human vision and touch. In the current implementation examples, images obtained from cameras are processed using AI ⁵. However, challenges still persist, such as measurement errors and camera-blind spots, which prevent the accurate capture of the exact moment of object grasping ⁶. The integration of force, temperature, and light sensing is crucial to overcome these challenges in agricultural environments. In this context, the light sensor functions as a proximity sensor to detect the distance to the target and its surface conditions before contact, thereby allowing the robot to decelerate and avoid high-impact collisions that could lead to bruising. Upon contact, the force sensor enables precise grasping force control. Simultaneously, the integrated heater and resistance thermometer detector (RTD) allow active thermal sensing to identify the thermal properties of the object. This capability is essential for distinguishing between the target crop and environmental obstacles, such as metal-supporting poles or plastic leaves, which may have similar visual appearances but vastly different thermal conductivities. This multimodal approach allows the robot to reliably identify and grasp crops even in fluctuating outdoor light or cluttered environments, where systems that are based on vision alone may fail. This study aims to realize such a multifunctional capability within a single miniature microelectromechanical system (MEMS) chip to provide a robust sensing solution for small robotic grippers. Prior works include commercially available sensors that combine any two of the following detection modalities: tactile, thermal, and proximity sensing ^7,8,9. However, these challenges are related to high production costs and difficulty in miniaturization.

Therefore, tactile sensors have been miniaturized using MEMS technology ^10,11. However, only a few studies have integrated multiple detection modalities. Consequently, we first attempted to combine force measurement and proximity sensing by integrating a light detection element using MEMS technology ¹². Subsequently, we integrated a temperature detection element to enable the combined measurement of thermal sensation ^13,14,15. This sensor is compact (chip size: 5 mm \(\times\) 5 mm) and is expected to achieve low cost through mass production.

In our previous studies, the combined measurement of force, light, and temperature was achieved using the same sensor. However, a drawback was the decrease in sensitivity and stability owing to the use of identical sensing elements for measuring temperature and light ¹⁶. Therefore, we fabricated the sensing elements separately. Furthermore, for light and temperature measurements, we employed DC resistance measurements instead of impedance measurement ^17,18. By implementing these methods, we achieved improvements in sensitivity and stability for both light detection and thermal sensation detection. Previous studies do not mention attempts made to integrate the three types of sensing elements, specifically the force sensing element and separately fabricated DC resistance-measurable temperature and light sensing elements, onto a single chip. In this study, we re-examined the layout and fabrication process of the light and thermal sensing elements and, by integrating them along with a force-sensing element, succeeded in performing simultaneous multimodal measurements of tactile, thermal, and proximity sensations at the same location. It was demonstrated that the contact state and characteristics of a contact object can be analyzed dynamically and in real time from the mechanical, thermal, and optical perspectives.

2. Tactile Sensor for Multi-Modal Measurement and Detection Principle

The detection principles for each sensing element are described below.

Regarding the principle of force detection, applying an external force to polydimethylsiloxane (PDMS) that encapsulates the cantilever causes the PDMS to deform. This deformation results in cantilever deflection. The resulting surface strain due to deflection was measured using a strain gauge. Therefore, an external force can be detected by measuring the strain gauge resistance.

Regarding the principle of thermal sensation detection, when there is a temperature difference between the sensor and the object, heat flow is generated upon contact. This causes a change in the temperature of the sensor, which is measured using the temperature dependence of electrical conductivity. Differences in the thermal conductivity and heat capacity of the objects which are in contact cause variations in the resulting change in the sensor temperature, making it possible to distinguish between perceived thermal sensations.

According to the principle of light detection, when light is incident on an object, the resistance changes because of the photoconductive effect. This change in the resistance enables the detection of light.

In previous studies, the conductivity, contingent on light and temperature, was measured through the change in the AC impedance of a circuit, facilitated by the insulating and active layers beneath the wiring. This method was prioritized because of its simplicity and ease of fabrication. However, this method suffers from issues related to parasitic capacitance and electromagnetic noise due to measurement at high frequencies. Furthermore, because the Si active layer retains its sensitivity to both temperature and light, it is difficult to separate the responses. As a countermeasure, we implemented DC resistance measurement by wiring the electrodes directly onto the Si layer for light detection and temperature measurements were made using the RTD of Au. Fig. 1 demonstrates the technical justification for adopting the DC measurement by comparing its stability with that of the conventional AC method. In this evaluation, the three times the standard deviation (\(3\sigma\)) value of the output signal was calculated over a fixed period as an index of response stability. For the impedance measurement, the \(3\sigma\) value was 0.43% for both the LED-off and LED-on states. In contrast, the DC resistance measurement achieved a significantly lower \(3\sigma\) value of 0.00022% in the unlit state and 0.00025% in the lit state. To validate the signal-to-noise ratio (SNR) comparison, it is crucial to consider the effective signal magnitude. It was confirmed that the fractional resistance change, \({\Delta R}/{R}\), under a constant light intensity, remained essentially unchanged regardless of the measurement method (AC or DC). For example, at an identical light intensity of 3120 lx, the response magnitudes of the two methods were comparable. Consequently, as shown in Fig. 1, the DC resistance measurement achieved a noise floor which was approximately 2,000 times lower than that of the impedance measurement while maintaining the same signal magnitude. This results in a significantly higher SNR, enabling the detection of minute resistance changes without being obscured by electrical noise.

Therefore, a Si photoresistor that uses DC resistance measurements for light detection was adopted. For thermal sensation detection, an Au RTD and a heater were integrated into the chip. This resulted in reduced noise and enabled stable measurements compared to impedance change measurements ¹⁸. Nakamura et al. fabricated a sensor that integrated a light-sensing element using a Si photoresistor and a thermal-sensing element using an Au RTD mounted on a single chip, and it could measure heat and light simultaneously and stably. However, they did not integrate a force-sensing element ¹⁹. Therefore, in this study, we designed a chip and its fabrication process, which integrates a force-sensing element (cantilever) in addition to the photoresistor and RTD. Fig. 2 shows the chip of the MEMS tactile sensor designed in this study. For light measurements, electrodes were fabricated to connect with the Si layer to form a photoresistor. For thermal sensation detection, a temperature resistor and heater were integrated into the chip. The on-chip heater allowed the sensor to reach thermal equilibrium faster than in the conventional method using an external heater, resulting in improved stability ¹⁸.

Figure 3 illustrates the sensor fabrication process. The sensor used in this study is fabricated using MEMS processing on an SOI wafer substrate. First, an insulating layer of Si\(_3\)N\(_4\) was deposited on an SOI wafer by RF sputtering. Subsequently, photolithography and etching were used to remove Si\(_3\)N\(_4\) from the areas designated as the Si photoresistor electrodes. Next, NiCr and Au thin films were deposited using RF sputtering, followed by patterning of the interconnects, strain gauge, and heater of NiCr and the interconnects of Au using photolithography and etching. The cantilevers were patterned in the same manner as described previously. The wafer was then separated into chips by dicing and a cantilever structure was formed through sacrificial layer etching. Finally, the chip was cleaned with deionized water, immersed in highly volatile ethanol to prevent stiction, and vacuum dried. Fig. 4 shows the sensor after implementation. The sensor chip was fixed onto a printed circuit board (PCB) using an epoxy adhesive and electrically connected via Au wire bonding. The MEMS structure and the bonding wires on the chip were encapsulated using spin-coated PDMS, and a hemispherical PDMS bump (\(\varphi\)2) was attached to the center of the chip. Additionally, an LED chip (1.3 mm \(\times\) 2.0 mm) was implemented on the PCB, for light measurement.

3. Multi-Modal Measurement of Force, Temperature, and Light Using a Tactile Sensor

3.1. Response Evaluation of the Temperature, Force, and Light

To perform combined measurements in a multimodal system, the response characteristics of each element to temperature, light, and force must be clarified. Because the strain gauge and photoresistor are also expected to exhibit temperature dependence, their respective temperature characteristics are measured.

Figure 5 shows the temperature dependence of (a) the Au RTD, (b) the NiCr strain gauge, and (c) the Si photoresistor. Due to the significant temperature-induced variation in the photoresistor, its resistance change was measured while the temperature was increased from 28°C to 31°C in increments of 1°C. Each sensing element exhibits a linear change in resistance with respect to temperature. For metals such as Au and NiCr, the resistance increases because of the enhanced scattering probability of free electrons caused by the increased atomic vibrations that accompany the temperature increase. Conversely, for semiconductors such as Si, the resistance decreases because the increase in carrier density accompanying the temperature increase outweighs the increase in electron scattering. The temperature coefficient of resistance (TCR) was determined to be 2415.8 ppm/°C for the Au RTD, 244.9 ppm/°C for the NiCr strain gauge, and \(-\)51776 ppm/°C for the Si photoresistor.

Figure 6 shows the force characteristics of the NiCr strain gauge. When a force is applied to the PDMS bump, the incompressibility of the PDMS causes it to be compressed vertically, resulting in horizontal expansion of the material. This horizontal expansion increases the deflection of the cantilever, inducing a compressive strain on its surface. Consequently, the resistance of the strain gauge decreases linearly with respect to the applied force. The force dependence of the NiCr strain gauge was determined to be \(-\)21.5 ppm/N.

Figure 7 shows the light characteristics of the Si photoresistor. When light is incident on a photoresistor, its resistance decreases linearly because of the photoconductive effect. The light dependence of the Si photoresistor was determined to be \(-5.54\times 10^{-5}\)%/lx.

3.2. Summary of Simultaneous Multi-Modal Measurement

Figure 8 shows the experimental setup for evaluating the response of the multimodal tactile sensor used in this study. The resistance of each sensing element is measured using a multimeter. A motorized stage was used to move the contact object along the \(z\)-axis. A six-axis force sensor was attached to the upper stage for contact force measurements for stage control. In this study, the contact object was mounted on the upper stage and the tactile sensor mounted on the PCB was fixed to the lower stage. An air gap is provided in the lower stage to mitigate the heat flow transferred when the sensor is heated. In this multimodal measurement, the resistance values of the strain gauge, photoresistor, and RTD were measured simultaneously.

The measurement procedure is outlined as follows. The \(z\)-stage was moved toward the sensor at a speed of 0.5 mm/s, starting from a position 50 mm away from the contact object to the top of the sensor’s PDMS bump. The normal force during contact was set to 10 N (with an error of \(\pm 10\)%). During measurement, the heater was heated by applying 10 V, and the sensor temperature stabilized at approximately 33.9°C before contact. The LED is illuminated by applying voltage of 2 V. The sensor was held in contact for 90 s to maintain a constant force of 10 N and then returned to its original position at the same speed. It should be noted that the measurements are conducted inside a dark box to prevent the influence of ambient light. Fig. 9 shows several contact objects, including Al, polyoxymethylene (POM), and PDMS, with different thermal, mechanical, and optical characteristics used for multimodal measurements. The Al and POM have holes drilled in their four corners for fixed mounting on a three-axis motorized stage using screws. Furthermore, PDMS was adhered to the surface of Al.

3.3. Results of Simultaneous Multi-Modal Measurement

Figure 10 shows the change in resistance of the Au RTD, NiCr strain gauge of the cantilever, and Si photoresistor for each contact object as a result of the multimodal measurement. The responses of the force and light sensors shown in Fig. 10 are the results obtained after applying temperature compensation to decouple the thermal influence from each sensing modality. The rate of change of temperature with respect to time \({\Delta T}/{T}\) is shown in Eq. \(\eqref{eq:1}\):

\begin{equation} \label{eq:1} \dfrac{\Delta T}{T}=\dfrac{\displaystyle\frac{\Delta R_{\textit{Au}}}{R_{\textit{Au}}}}{\alpha_{\textit{Au}}}, \end{equation}

where \({\Delta R_{\textit{Au}}}/{R_{\textit{Au}}}\) denotes the fractional resistance change based on the resistance immediately before contact, and \(\alpha _{Au}\) denotes the TCR of the RTD. The sensor temperature calculated using this equation is shown on the right side of the temperature measurement graphs. Because the force- and light-sensing elements are affected by temperature variations, temperature compensation is necessary. Temperature compensation was performed using the temperature change measured by the RTD (shown in Fig. 5) and temperature dependence of each resistor, which is expressed in Eq. \(\eqref{eq:2}\):

\begin{equation} \label{eq:2} \dfrac{\Delta {R_{\textit{TC}}}'}{{R_{\textit{TC}}}'}=\dfrac{\Delta R_{\textit{TC}}}{R_{\textit{TC}}}-\dfrac{\Delta T}{T}\times \alpha_{\textit{TC}}, \end{equation}

where \({\Delta R_{\textit{TC}}}/{R_{\textit{TC}}}\) and \(\alpha_{\textit{TC}}\) denote the fractional resistance change and the TCR of the element to be compensated, respectively. By subtracting the thermal components from the real-time RTD values, the independent responses of force and light were successfully extracted without being buried in the thermal drift.

4. Discussion

As shown in the temperature measurement graphs, the RTD resistance decreases upon contact and returned to its initial value when the two surfaces are separated. Because Al has a higher thermal conductivity than POM, the decrease in resistance is greater. Although PDMS has a thermal conductivity similar to that of POM, it exhibits a greater decrease in resistance, which is considered to be related to the increase in contact area caused by the difference in hardness and heat transfer to the Al object on which the PDMS is adhered. Furthermore, in the case of POM and PDMS, the temperature initially decreases and then begins to increase. This is considered to be due to the fact that heat transfer is influenced not only by differences in thermal conductivity but also by density, specific heat capacity, and surface shape, or because convective heat transfer is obstructed by the presence of the object ^16,18.

The force measurement graphs show that the resistance of the strain gauge decreased upon contact, demonstrating the application of force. Because of the temperature compensation, the strain gauge resistance after contact becomes nearly constant over time, correctly demonstrating the state in which the sensor is held stationary under a constant force. However, the magnitude of the change in resistance differed among the contact objects. Although the target load was set to 10 N, the actual applied loads monitored by the six-axis force sensor showed slight variations within a tolerance of \(\pm 10\)%, specifically 10.67 N for Al, 10.15 N for POM, and 10.24 N for PDMS at 50 s after contact. The proposed sensor was sensitive enough to detect these 0.1 N-level differences as distinct fractional resistance changes. Additionally, these differences are influenced by the variation in the force dependence of the strain gauge, as shown in Fig. 6. Furthermore, in the case of the PDMS, the resistance gradually increases after initially decreasing. This is attributed to the relaxation in stress caused by the viscoelasticity of PDMS, which leads to redistribution of the internal stress after the initial contact.

The light measurement graphs show that, as the object approaches, the resistance of the photoresistor decreases owing to an increase in the reflected light from the contact object. Although the specific surface roughness of the test samples was not measured in this study, the materials were selected to represent the distinct optical reflection properties as a baseline evaluation. The magnitude of the reflected light varies depending on the material properties of the target; for materials with large surface roughness, such as standard machined POM, diffuse reflection increases the total amount of light incident on the photoresistor. In contrast, for glossy materials such as commercially available smooth Al plates, specular reflection is dominant, which results in a large reflection only at specific angles, potentially reducing the overall amount of light incident on the sensor compared with diffusing surfaces. Furthermore, internal reflection within the PDMS is a critical factor in the change in resistance when a force is applied. The light incident on the sensor includes not only reflections from the contact surface, but also those from the side and top surfaces, including the adhesion interface between the PDMS and the Al frame. This internal optical network is sensitive to the geometric changes of the sensor body. The application of force upon contact compresses the elastomeric body and pushes the PDMS closer to these reflective surfaces, consequently decreasing the optical path distance and increasing the amount of incident light captured using a photoresistor. After the contact was established, the resistance of the photoresistor became nearly constant. This stability occurs because the relative distance between the sensor and the object remains fixed during the stable grasping phase. Moreover, by applying the described temperature compensation, the influence of the heat generated by the integrated heater or environment is effectively canceled, allowing the sensor to provide a steady signal that clearly indicates the onset of a stable grasp and ensures the decoupling of thermal and optical stimuli.

5. Conclusions

This study successfully designed and fabricated a multimodal MEMS tactile sensor capable of dynamic and simultaneous measurement of force, thermal sensation, and light, satisfying the requirements for delicate agricultural robotic tasks, such as a force resolution of 0.1 N and rapid response. For light detection, the implementation of a Si photoresistor using DC resistance measurement effectively minimized electromagnetic noise, achieving an exceptionally high \(3\sigma\) stability of 0.00022% compared to conventional AC-based methods. For thermal sensing, the on-chip integration of a Au RTD and a heater significantly improved both the response speed and measurement stability. Furthermore, by utilizing the real-time RTD values as a reference for temperature compensation, the thermal influences on the strain gauge and photoresistor were successfully decoupled, allowing for independent extraction of each modality within a compact 5 mm \(\times\) 5 mm footprint.

The developed measurement system operates at a sampling rate of 0.1 s, providing a temporal resolution of 10 Hz. In this study, the term dynamic refers to the capability of real-time sequential monitoring of grasping phases from the approaching phase to the stable contact phase. At typical approach speeds for agricultural grippers (10–50 mm/s), this resolution enables data acquisition for every 1–5 mm of movement, which is sufficient to perform feedback control, such as decelerating the gripper before contact to minimize impact or adjusting the grasping force to prevent slip ²⁰. Therefore, the developed sensor and its measurement system demonstrated adequate responsiveness and a superior balance between miniaturization and measurement reliability for the intended agricultural applications.