1. Introduction

Surface inspection plays a crucial role in quality control within manufacturing processes, as it aims to detect defects present on product surfaces. In automated surface inspection, non-contact optical methods—such as camera-based sensing—are generally preferred to ensure that the inspection process itself does not affect product quality ¹. However, when the defects to be detected cannot be distinguished from normal regions using non-contact methods, alternative inspection techniques become necessary. For example, in the food manufacturing industry, determining whether all shells have been removed from raw shrimp is difficult with camera-based inspection, so human inspectors typically rely on visual observation and tactile feedback to identify remaining shells. In particular, when the goal is to detect hard foreign objects mixed within soft food products, tactile, contact-based inspection is often highly effective.

Tactile sensing in humans is distributed in an array-like manner to capture the position and shape of contact areas. A tactile sensor with performance comparable to that of a human fingertip should provide a spatial resolution of approximately 1 to 2 mm and incorporate 50 to 100 sensing points ². Furthermore, human tactile perception can detect height variations on the submillimeter scale. Depending on the type of product, certain inspection tasks rely on the extremely high tactile sensitivity of the human hand. To automate such inspection tasks, a tactile sensor with both high spatial resolution and high sensitivity is essential ³, and the development of various tactile sensors and their applications have been studied ^4,5,6.

Tactile sensors capable of obtaining high spatial resolution and rich information using a camera have been proposed. These sensors are commonly referred to as tactile image sensors or vision-based tactile sensors ^7,8,9. A variety of tactile image sensors have been developed to date, and new designs continue to be proposed. Our research group has developed several tactile image sensors, which we have primarily employed for robotic hand grasping control and manipulation tasks ^10,11. Tactile image sensors generally consist of a camera and a tactile skin that converts physical contact into an optical signal detectable by the camera. Depending on how contact is converted into optical information, these sensors can be classified into several types ¹². For example, the marker displacement method is one of the most widely used approaches, in which a camera measures the displacement of visual markers embedded in a soft material ^13,14,15. This method is particularly suitable for detecting the magnitude and direction of applied forces. In the reflective membrane method, a camera captures small deformations on the surface of a flexible sheet covered with a reflective membrane ^16,17. Because minute irregularities on the sensor surface are emphasized by illumination applied from the side, even very small surface features, such as human fingerprints, can be visualized. Reflective membrane-type tactile image sensors thus have the potential to detect extremely small defects and are particularly suitable for contact-based surface inspection.

In order to efficiently perform contact inspection of the object to be inspected, it is necessary to consider the shape and configuration of the sensor. Fig. 1 shows typical sensor shape and configuration. Most tactile image sensors proposed to date have been designed primarily for robotic manipulation and are therefore intended to be mounted on the fingertips of robot hands ²¹. Consequently, they are typically compact and feature relatively flat contact surfaces. In recent years, sensors with curved geometries that more closely resemble the human fingertip have also been developed ^22,23,24. For inspection-oriented applications, however, the use of such fingertip-type sensors requires the sensor to be moved up and down repeatedly to sequentially touch multiple inspection points one by one. As a result, when inspecting a large surface area or a large number of objects, this approach becomes time-consuming, and these sensor shapes are not necessarily efficient. To address this issue, a roller-type tactile image sensor was proposed, in which a camera is placed inside a cylindrical sensing surface ^18,19. This sensor geometry is well suited for surface inspection, as it enables rapid, continuous inspection by rolling the sensor surface across large inspection areas or over surfaces where many objects are arranged. Furthermore, a belt-type tactile image sensor was proposed, in which a looped belt-shaped sensing surface is rotated by two wheels ²⁰. In the roller-type sensor, the contact area at any given moment is relatively small; therefore, when the defect pattern on an object’s surface needs to be captured, it is necessary to roll the sensor to scan the surface while generating a composite image. Because the belt-type sensor provides a larger contact area, it enables efficient use of the camera’s field of view and allows direct visualization of surface height or defect patterns without requiring image compositing.

All of the tactile image sensors described above can inspect only one side of an object—the side that comes into contact with the sensing surface. For plate-like objects such as printed circuit boards or fiber sheets, or for granular food items such as shrimp or shellfish, defects located on the backside cannot be detected unless the object is flipped over by some means. This requirement introduces an additional and cumbersome step in automated inspection processes.

In this study, we developed a double-roller tactile image sensor in which two roller-type sensors are arranged facing each other to enable simultaneous inspection of both the front and back surfaces of a target object (Fig. 2). This configuration allows simultaneous detection of defects on both sides of plate-like objects such as printed circuit boards, as well as the simultaneous inspection of foreign materials on both sides of food items such as shrimp.

2. Double Roller Tactile Image Sensor

2.1. Tactile Sensing Principle

A tactile image sensor consists of a sensing surface that comes into contact with the target object, a camera that captures the sensing surface from behind, and a computer that analyzes the camera images to extract tactile information. The sensing surface includes a mechanism that converts the physical contact of an object into optical information so that the contact-induced changes can be observed in the camera images. The spatial and temporal resolution of the tactile sensor is essentially determined by the specifications of the camera and the performance of the image-processing algorithms.

Various methods have been proposed for converting physical contact on the sensing surface into optical information ⁷. In this study, we adopt the reflective-membrane method, which is particularly well suited for inspection applications ^16,17. The structure of a tactile imaging sensor based on this reflective-membrane method is shown in Fig. 3. The sensing surface that contacts the object is made of a transparent, soft elastic sheet. A thin reflective membrane is formed on the outer surface of this sheet. The reflective membrane serves to reflect light from a light source and generate shading that corresponds to the surface’s minute height variations. Ideally, the membrane should be uniform in color and as thin as possible; for example, it can be created by coating the surface with a thin layer of silver paint.

When an object contacts the surface of the elastic sheet coated with a reflective membrane, the soft material deforms according to the surface geometry of the object. The transparent elastic sheet is bonded to a light-guiding plate (e.g., a transparent acrylic plate) that incorporates illumination sources along its edges, forming an edge-lit light-guiding illumination system. Light emitted from the sources propagates through the light-guiding plate and the transparent elastic layer, illuminating the reflective membrane from behind. The camera captures the resulting deformation pattern of the reflective membrane, which is emphasized by variations in surface height and therefore becomes clearly visible in the images. This configuration enables the capture of fine surface features; for example, the system can visualize the ridges and valleys of a human fingerprints.

2.2. Structure of the Sensor

Figure 4 shows the structure of the two roller-type sensors that constitute the double-roller tactile image sensor. The roller-shaped sensing surface is constructed by attaching a urethane-gel sheet (Asker C23, thickness 5 mm) onto an acrylic cylinder (length 150 mm, thickness 3 mm), and then coating the outer surface with silver paint to form a reflective membrane. In addition, a transparent film dressing (Tegaderm, 3M) is applied to protect the reflective membrane. The outer diameter of the roller is 80 mm. An aluminum hollow shaft (outer diameter 24.96 mm) is mounted along the rotational axis of the roller, and both a white LED and a camera are installed on this shaft. The camera used is a CMOS model, IDS UI-3481LE-M-GL (\(2560 \times 1920\) pixels, monochrome). The camera cable and LED power cable pass through the inside of the hollow shaft and can be routed to the outside of the roller. The black resin components that fix the camera to the side cover and shaft are fabricated by laser-cutting acrylic plates and assembling them through stacking and bonding. The black side cover contains an O-ring, which is held in place by the friction generated between the O-ring and the roller. Bearings are also implemented as axle supports to reduce friction between the cover and the shaft.

Figure 5 illustrates the overall structure of the double-roller sensor. Two rollers with the same construction described above are fixed onto an aluminum frame. One of the rollers is mounted at the end of the aluminum frame, while the position of the other roller can be adjusted manually to change the spacing between the two rollers. The fixed roller serves as the driving roller and is rotated by a servo motor. The driving roller and the servo motor are equipped with spur gears, which transmit the rotation of the servo motor to the roller. The spur gear on the roller side is made of polyacetal resin and has 68 teeth. The servo motor side consists of the servo motor, a planetary gear set, and a spur gear. The servo motor used is KRS-3301 ICS (Kondo Kagaku Co., Ltd., maximum torque 4.9 kg\(\cdot\)cm, maximum speed 62.5 rpm). Two planetary gear units with a \(4:1\) gear ratio are stacked to achieve an overall \(16:1\) reduction. The spur gear on the servo motor side is also made of polyacetal resin and has 34 teeth.

3. Evaluation of the Sensitivity

3.1. Experimental Setup

The double-roller tactile image sensor acquires tactile images by grasping an object from both sides with two rollers and applying force. Understanding how the sensor’s response varies depending on the applied force and the properties of the object allows us to determine the applicable range of the sensor. To this end, we investigated the relationship between the thickness of foreign particles, the hardness of the object, and the magnitude of the applied force, simulating a situation in which foreign objects are present on the surface of the target.

Figure 7 shows the experimental setup. In the experiment, one roller-type sensor was used, and objects simulating small foreign particles were pressed against its surface using a force gauge. The test objects were nylon threads with diameters of 0.25 mm and 0.9 mm, each 10 mm in length. The tip of the force gauge was equipped with a urethane gel layer (thickness 5 mm, hardness Asker C23) identical to the roller surface, and a test material (thickness 5 mm) simulating the inspection target was attached on top. Test materials of different hardnesses were used, including Asker C0, Asker C23, and a hard acrylic plate, to compare the sensor’s response across a range of material stiffness.

4. Inspection of Both Front and Back Surfaces of Food

We investigated the feasibility of applying the developed double-roller tactile image sensor to foreign object detection in actual food products. As the test specimen, we used raw shrimp, as shown in Fig. 9. Two types of shrimp were prepared: one with the shell completely removed and another with a portion of the shell intentionally left on one side, which served as the foreign object to be detected. The shrimp had a maximum thickness of approximately 13 mm, and the remaining shell portion measured approximately 9 mm in thickness. The shrimp were inserted from above into the gap between the two rollers for the experiments. Measurements were conducted for two roller gaps: 8.0 mm and 9.5 mm.

Figure 10 shows the tactile images simultaneously acquired from the top and bottom sides of the shell-free and shell-attached shrimp using the double-roller tactile imaging sensor. Images obtained with roller gaps of 9.5 mm and 8.0 mm are presented in the upper and lower rows, respectively. The green and red lines in Fig. 10(a) indicate the positions where the pixel value profiles (Figs. 10(b)–(e)) were acquired. Regarding slippage between the roller and the shrimp in this experiment, there was almost no difference in the length of the object on the output image between the four different experiments, which indicates that no significant slippage occurred. For each image, the horizontal (\(X\)) and vertical (\(Y\)) intensity profiles corresponding to the region with the remaining shell are shown in the middle and bottom graphs. In each graph, the profiles for the shell-free and shell-remained shrimp are plotted in green and red, respectively. Figs. 10(b) and (d) are plots of pixel values along the \(Y\) coordinate at a specific \(X\) position indicated in Fig. 10(a), with a 10-pixel moving average in the \(Y\) direction applied to remove high frequency image noise. Similarly, Figs. 10(c) and (e) are plots of pixel values along the \(X\) coordinate at a specific \(Y\) position indicated in Fig. 10(a), with a 10-pixel wide moving average in the \(X\) direction. In Figs. 10(b), (c), and (e), only the red solid line (Sensor 1 for the shrimp with shell) shows a large response at the shell region. This means, for both roller-gap conditions, the images of the shrimp with shell exhibit regions of elevated intensity that do not appear in the shell-free images, indicating the potential for detecting the remaining shell. In particular, when the roller gap is 9.5 mm, the response at the shell region is significantly larger than at any portion of the shell-free shrimp, making it possible to easily detect as a foreign object.

However, when the roller gap is reduced to 8.0 mm, although the shell region still produces a strong response, the thicker portion of the shrimp (upper area of the tactile image) also produces a similarly large response (Fig. 10(d)). This occurs because the greater thickness of the shrimp increases the force with which the roller sensor is pressed against it. Conversely, the tail region (lower area of the tactile image), which is relatively thinner, experiences a weaker pressing force and therefore yields a smaller response. These results indicate that selecting an appropriate roller gap according to the thickness of the object enables reliable detection of foreign objects such as shrimp shells.

In actual image processing for this foreign object inspection, a uniform process is applied to the entire image to detect foreign objects. For example, the raw image obtained from the tactile sensor is binarized based on thresholding, followed by detection of connected components (blobs) and removal of minute regions based on their area, resulting in the detection of the remaining connected components as foreign objects. Introducing detection methods based on deep learning might also be effective. Developing detection algorithms tailored to specific foreign objects will be a challenge for each application.

5. Conclusion and Discussion

In this study, we developed a double-roller tactile image sensor that combines two roller-type sensors to simultaneously inspect both the top and bottom surfaces of an object. The sensor employs a reflective-membrane structure, enabling visualization of fine surface irregularities. The object is placed between the two roller sensors, and tactile images are continuously acquired while the rollers rotate. These images are then stitched together to generate a tactile image of the entire object surface. Sensitivity experiments demonstrated that a thin nylon thread with a diameter of 0.25 mm placed on a soft specimen with Asker C0 hardness could be detected when a pressing force of at least 0.1 N was applied. Furthermore, we evaluated the sensor’s response to remaining shell fragments on raw shrimp. By appropriately setting the gap between the two rollers, the shell fragments could be selectively detected.

On the other hand, in the current design, the positions of the two roller sensors are fixed, so to achieve the appropriate pressing force, the distance between the two rollers must be set appropriately depending on the thickness of the target object. For objects with a constant thickness, such as a printed circuit board, the appropriate roller distance can be selected in advance. However, if the thickness of the object changes, the pressing force applied by the rollers will vary depending on the location, which will affect the response amplitude. In the shrimp-shell detection experiment, not only the shell fragments, which are the intended foreign objects, but also the thicker portions of the shrimp produced similarly large responses. To address this issue, it is desirable to introduce a mechanism that maintains a constant pressing force between the two rollers regardless of object thickness. For example, a slider mechanism combined with a spring that generates a constant load may be used. Also, in the present configuration, only one of the rollers is driven by a motor, while the other roller rotates through frictional contact with the object. Although sufficient friction was available in the experiments conducted here, problems may arise when the object is slippery and the friction force is insufficient. Driving both rollers using gears or similar mechanisms would improve the robustness and versatility of the system. Furthermore, if a mechanism that can actively control the roller spacing is introduced, it may be possible to estimate mechanical properties such as the hardness of an object by continuously changing the roller spacing and analyzing tactile images at different contact forces. As another challenge, expanding the range of objects to be inspected is a practically important issue. While increasing the length of the rollers would be beneficial, it is necessary to consider methods for achieving a uniform contact pressure distribution across the width. It may also be necessary to increase the number of cameras to cover a wider contact area. In a proposed tactile image sensor integrated into a robot link, cameras are mounted at both ends of a cylinder to capture its interior, and this design could serve as a useful reference ²⁵.

In future work, we will evaluate whether the inspection speed and actual detection performance satisfy the practical requirements of real product inspection.