3.1. Problem Setting

MGP is a global palpation approach designed to narrow down the search area for subsequent local palpation. The inputs for MGP are the indentation depths of the respective pressing patterns, while the output is the estimated probability distribution of the inclusion’s location. Local palpation is then directed only to the grid cells with a high probability of containing an inclusion, thereby reducing the total number of palpation trials. Therefore, in this fundamental validation of MGP, we focus primarily on identifying the approximate location of the inclusion, rather than characterizing its shape, size, or depth. As shown in Fig. 2, the target object is a square silicone phantom divided into a \(5 \times 5\) grids. Since exhaustive local palpation alone would require a maximum of 25 trials to locate the inclusion, we define the criterion for success as achieving identification with a combined total of fewer than 25 trials (MGP plus local palpation). However, for the purpose of evaluating MGP’s performance, physical local palpation is not actually executed; instead, the total number of trials required to locate the inclusion is calculated virtually based on the MGP estimation results.

3.2. Correspondence Between GI and MGP

In GI, a pattern with spatially random luminance is irradiated against the target from the projector. Since the reflectance and transmittance of the target vary along its surface, the reflected intensity measured by a single-pixel sensor differs with each pattern. By taking advantage of this phenomenon, the image of the target can be reconstructed by computing the correlation between the light intensity detected by the single-pixel sensor and the pattern ^19,20. Denoting by \(I_n(x,y)\) the light intensity distribution of the projector’s \(n\)-th irradiation and by \(B_n\) the reflected light intensity measured by the single-pixel sensor, the correlation distribution of the deviation of light intensity, \(G(x,y)\), can be written as:

In MGP, the pressing pattern is mounted at the end of a probe, which presses it against the target according to preset reference values. When an inclusion is present in the target, the pressing depth varies depending on the arrangement of convex shapes on the pressing pattern. The pressing depth at this time is measured using the encoder housed in the probe. This operation is repeated \(n\)-th times by changing the pressing pattern attached to the probe. Denoting by \(\mathrm{P}_n(x,y)\) the shape of the pattern used in the \(n\)-th trial and by \(D_n\) the pressing depth measured using the encoder, the correlation distribution of the deviation of pressing depth, \(G_{\rm{p}}(x,y)\), can be expressed:

3.3. Extension of GI to MGP

MGP and GI have two major differences. The first is that the pressing depth depends not only on the relative positions of the inclusion and convex shapes of the pressing pattern but also on the structure of the pressing pattern. The second is that the pressing pattern entails a large fabrication cost and a manual replacement task.

With regard to the first issue, the pressure exerted by the pressing pattern on the target is lower when there are a greater number of convex shapes. As a result, the pressing depth is reduced. Thus, a correlation exists between the pattern’s structure besides the positions of the convex shapes and the pressing depth. It then becomes difficult to estimate the location of the inclusion on the MGP stiffness map when this correlation is large.

With regard to the second issue, GI readily allows the production of various patterns since they can be modified by software changes. Meanwhile, MGP requires fabrication of the patterns in advance, which must be manually replaced to switch patterns, so that there is a limit to the number of patterns available for measurement.

To address these issues, we rotate the probe end so that the number of patterns used for measurement can be reduced. By employing the same pattern for different trials, it is expected to lower the effect of the correlation between pattern structure and pressing depth.

3.3.3. Fabrication and Selection of Pressing Patterns Based on Consideration of Rotation

We first describe the fabrication of pressing patterns that satisfy performance requirements 1 and 3, then discuss the method of assessment of performance requirement 3 and coordination requirement 1.

Arrangement of rotationally asymmetric patterns with fixed number of convex shapes (performance requirements 1 and 3)

Basic procedure: The pressing pattern’s cells are divided into subsets, where each subset corresponds to the pressing of specific points on the target.

The pressing patterns are fabricated in groups according to these subsets. We define the set of all possible positions that points on the surface of a given shape can occupy when subjected to a rotational operation as the convex-concave orbit (CC orbit). The CC orbits represent the subsets described above. For example, when a square sectioned into a \(5\times5\) matrix is incrementally rotated by 90°, as shown in Fig. 3(a), the 25 cells each describe one of seven CC orbits. Denoting by \((x, y)\) the coordinates of the \(x\)-th row and \(y\)-th column, CC orbit 2 is a subset consisting of four points, \((1, 2), (2, 5), (5, 4)\), and \((4, 1)\). CC orbit 0, at the center of the matrix, is the subset consisting only of \((3, 3)\). As this example shows, the number of cells that describe the CC orbit at the rotation center may be singularly different from those that describe other orbits. The singular CC orbit, which is denoted by CC orbit 0, is treated differently from the other CC orbits. We now introduce variables \(^{i}n_{\rm{conv.}}\) and \(a_j\) to the CC orbits, to formulize performance requirements 1 and 3. \(^{i}n_{\rm{conv.}}\): the number of convex shapes that describe CC orbit \(i\), \(a_j\): the number of CC orbits for which there are \(j\) convex shapes, where CC orbit 0 is excluded from \(a_j\). As shown in Fig. 3(b), for instance, the singular CC orbit 0 is included in the seven CC orbits, so that \(\sum_{j=0}^{4} a_{j} = 6\). Furthermore, CC orbit 0 consists of a single convex shape, CC orbit 1 of three convex shapes, while the other orbits have no convex shapes. Using the variables, this can be expressed as \(^0n_{\rm{conv.}} = 1\), \(^1n_{\rm{conv.}} = 3\), \(^2n_{\rm{conv.}} \sim \;^6n_{\rm{conv.}} = 0\).

For the pressing pattern to be rotationally asymmetric, at least one among \(a_1\), \(a_2\), or \(a_3\) must be nonzero. For a \(5 \times 5\) square, performance requirements 1 and 3 can be expressed as

\begin{align} \begin{cases} {}^{0}n_{\rm{conv.}} + \displaystyle\sum_{i=0}^{4} i a_{i} = N_{\rm{conv.}} = \rm{Const.}, \\ a_1 + a_2 + a_3 \neq 0, \end{cases} \end{align}

where \(N_{\rm{conv.}}\) is the number of convex shapes on the pressing pattern.

Selection of patterns with high rotational asymmetry and low correlation: (a) Coordination requirement 1, (b) performance requirement 3

Basic procedure: The correlation between patterns is assessed by the inner product of vectors expressing arrangements of convex shapes on the pressing pattern. Rotation is expressed by shifting the vector components.

As shown in Fig. 3(c), the pressing pattern \(\rm{P}_\mathit{n}\) can be expressed by a vector \(\boldsymbol{V}_n \in \mathbb{R}^{1 \times 25}\), which consists of vector subspaces, each of which represents cells that describe a given CC orbit. Denoting by \(c^i_{k}\) the presence or absence of a convex shape at the \(k\)-th element of CC orbit \(i\), we can write

\begin{equation} \boldsymbol{V}_n =\left[ \begin{array}{c|c|c} c^0_1, &c^1_1,c^1_2,c^1_3,c^1_4,&c^2_1,c^2_2,c^2_3\dots \end{array} \right], \label{eq:pattern_vector} \end{equation}

where \(c^i_{k}=0\) when that element contains a convex shape and \(c^i_{k}=1\) when it does not contain a convex shape.

(a) Evaluation of and selection based on coordination requirement 1

The correlation \(R_{\rm{pattern}}\) between pressing patterns \(\rm{P}_\mathit{n}\) and \(\mathrm{P}_{n+1}\) can be written as

\begin{equation} R_{\rm{pattern}} = \boldsymbol{V}_n \cdot \boldsymbol{V}_{n+1}. \label{eq:pattern_correlation} \end{equation}

We select pairs of pressing patterns for which the correlation \(R_{\rm{pattern}}\) is small.

(b) Evaluation of and selection based on performance requirement 3

Next, in order to increase the number of pressing patterns based on rotation, we select patterns which display a low correlation between the original and rotated ones. When the figure possesses \(l\)-th order rotational symmetry, the minimum rotation angle is \({360}/{l}\) [°], so that rotation of the pattern can be expressed by shifting element \(c^i_{k}\) of each CC orbit \(i\) by one position. The vector \(^{\theta \rm{deg.}}\boldsymbol{V}_{n}\), obtained by rotating \(\boldsymbol{V}_n\) by \(\theta\) [°], is then expressed as:

\begin{equation} ^{\theta \rm{deg.}}\boldsymbol{V}_{n} =\left[ \begin{array}{c|c|c} c^0_1, &c^1_4,c^1_1,c^1_2,c^1_3,&c^2_4,c^2_1,c^2_2,\dots \end{array} \right], \label{eq:matrix_rotation_theta} \end{equation}

which, compared to Eq. \(\eqref{eq:pattern_vector}\), shows that the elements of CC orbits 1 and 2 have been shifted to the right by one position, while the last element has been moved to the front. The remaining CC orbits, 3–6, of \(^{\theta \rm{deg.}}\boldsymbol{V}_{n}\) are shifted similarly. The correlation \(R_{\rm{rotation}}\) between two pressing patterns before and after rotation can be expressed using vectors \(\boldsymbol{V}_n\) and \(^{\theta \rm{deg.}}\boldsymbol{V}_{n}\), as follows:

\begin{equation} R_{\rm{rotation}} = \boldsymbol{V}_n \cdot ^{\theta \rm{deg.}}\boldsymbol{V}_{n}. \label{eq:rotation_correlation} \end{equation}

We select pressing patterns for which this correlation \(R_{\rm{rotation}}\) is small.

The above method of producing pressing patterns using the CC orbits and assessing their correlation by employing lower-dimensional vectors is applicable to shapes other than squares, as shown in Fig. 4.

4.1. Simulation Conditions

The simulation conditions of the target object and pressing pattern are presented in Table 1. The target consists of a square with the dimensions \(X= 100\) pixels wide and \(Y=100\) pixels long, which is sectioned into a \(5\times5\) array. A circular inclusion is placed at the center of a randomly selected cell. In the simulation, measurement of the pressing depth is idealized, where the depth takes the value 0 when a convex shape comes into contact with the inclusion and the value 1 when it does not come into contact with the inclusion. Furthermore, the deviation between the centroid of the convex shapes and the probe’s point of application is not taken into account. The detection accuracy of the inclusion is evaluated from the mean squared error (MSE) of its position in the target and the stiffness map, as follows:

4.2. Simulation Results

A pressing pattern with \(N_{\mathrm{conv.}}\) convex elements and a pressing pattern with \(25 - N_{\mathrm{conv.}}\) convex elements (i.e., \(N_{\mathrm{conv.}}\) concave elements) exhibit fundamentally identical detection performance, because the presence or absence of contact between the inclusion and the convex elements is simply reversed. For this reason, the evaluation range for \(N_{\mathrm{conv.}}\) was set from 1 to 13. The simulation results are shown in Fig. 5. It can be seen that the MSE of the red curve, representing \(N_{\mathrm{conv.}}=13\), the number of convex shapes, falls rapidly with the number of presses. Due to the characteristic of correlation operations, the stiffness map does not display information regarding the location of the inclusion when contact occurs with the inclusion at every press and when no contact occurs for every press. In other words, the detection accuracy suffers due to lack of information about the entire target when \(N_{\mathrm{conv.}}\) is low and there exists a bias on the position of the convex shape on the pressing pattern. In the case of \(N_{\mathrm{conv.}}=13\), when roughly one-half of the cells represent convex shapes, there is a 50% probability of contact at each press, which is why information is acquired efficiently. Regarding the number of palpation trials, the rate of decrease in the MSE diminishes after the eighth trial. We attribute this to the fact that setting \(N_{\mathrm{conv.}} = 13\) creates conditions similar to a binary search. However, because the pressing patterns are generated randomly in this simulation, more trials are required compared to the theoretical minimum of five trials expected in an ideal binary search. Therefore, in simulation 2, we employ two patterns which are varied by rotating them, each of which is pressed eight times.

The results of simulation 2 are shown in Fig. 6. In this simulation, we randomly fabricated pressing patterns under the limitation that they must be rotationally asymmetric. In order to narrow down the selection of pressing patterns to satisfy performance requirement 3 and coordination requirement 1, we first generated 100 pairs of patterns, then selected from among them pairs of patterns that displayed the lowest correlations of Eqs. \(\eqref{eq:pattern_correlation}\) and \(\eqref{eq:rotation_correlation}\). In the simulation, the following three conditions were compared.

• Elite: Two pressing patterns that were selected by the above process.

• No-elite: Two pressing patterns that were not subjected to any selection process.

• Rand.8: Eight patterns randomly selected and not increased by rotation.

In terms of the effect of increasing patterns by rotation, we found a significant difference between the final MSE values of non-elite and rand.8, indicating that simple rotation lowers the detection accuracy. Meanwhile, the rate of decline and final values of MSE of elite and rand.8 were about the same, which suggests that the effect of increasing patterns by simple rotation can be offset by evaluating the correlation and subjecting the patterns to a strict selection process.

5.1. Experiment Method

The experiment setup is shown in Fig. 7(a). To press the pressing pattern using a probe, we employed a shaft motor (SL083, Nippon Pulse Motor Co.) mounted to a robot arm (FR5, FairINO). The pressing pattern, with the dimensions \(48\times 48\) mm, was fabricated using a 3D printer. The target consists of a \(70\times 70 \times 9\) mm silicone-gel block (stiffness 0, Exseal Co.), of which the central area was used as the search range for inclusions. The shaft motor was driven by PD control (\(K_{\rm{p}} = 0.736\) N/mm, \(K_{\rm{d}} = 0.400\) Ns/mm) to follow a fifth-order polynomial \(z_{\rm{ref}}\), to press the pressing pattern. Note that the boundary conditions for the fifth-order polynomial are \(z_{\mathrm{ref}}(0)=z_{\mathrm{ref}}(10~\mathrm{s})=1\) mm, and \(\dot{z}_{\mathrm{ref}}(0)=\dot{z}_{\mathrm{ref}}(10~\mathrm{s})=\ddot{z}_{\mathrm{ref}}(0)=\ddot{z}_{\mathrm{ref}}(10~\mathrm{s})=0\) mm/s.

5.2. Correlation Between Pressing Depth and Deviation Between Convex-Shape Centroid and Point of Application

The probe’s point of application is fixed at the center of the pressing pattern. We measured the pressing depth using multiple patterns with different convex-shape centroids, thereby varying the deviation of the point of application and centroid. We used Obj.0, shown in Fig. 7(b), with no inclusions, as the target and the pressing patterns shown in Figs. 7(c) and (d1). For each pressing pattern with a different convex-shape centroid, we carried out four presses and measured the pressing depths, which are shown in Fig. 8. The correlation coefficient between the deviation and pressing depth was found to be \(0.56\), where the results displayed the tendency of increasing pressing depth with greater deviations. This finding suggests the need to limit the use of pressing patterns in MGP to those for which the deviation between the convex-shape centroid and probe point of application is small.

5.3. Verification Experiment of MGP

We conducted an experiment to verify the feasibility of using MGP for global palpation. As targets, we used two blocks which contained inclusions, as shown in Fig. 7(b). Obj.1 has a single inclusion while Obj.2 has two inclusions. The pressing patterns used are those shown in Figs. 7(d1) and (d2). These patterns were selected from those which were successful in locating the inclusion in simulation 2, described in Section 4.2, but in which the convex-shape centroid and probe application point are aligned.

In order to realize MGP, it is necessary for the pressing depth to vary depending on the number of inclusions which have come into contact with the pattern. Table 2 presents the results of the pressing depth of the patterns according to the number of inclusions (num. hits) that have come into contact with the pattern. It can be seen that the pressing depth decreases as the number of inclusions in contact with the convex shapes increases, indicating that the basic performance necessary for MGP is satisfied. The pressing depths of Table 2 are sufficiently above the measurement limit since the encoder’s resolution is 5 . Furthermore, in light of the experimental conditions described in Section 5.1, the influence of vibrations and deflection within the support system is considered to be negligible.

Next, we constructed a stiffness map from the results of using two pressing patterns rotated to increase patterns, so that they were pressed a total of eight times. This map can be viewed as the global palpation results using MGP, and indicates the priorities to carry out local palpations. The stiffness map based on MGP was evaluated based on the following two indices. First hit/trial \(=\) the number of local palpations necessary to locate an inclusion after local palpation has been initiated. Max palpation/trial \(=\) the maximum number of palpations necessary to palpate the second inclusion after local palpation has been initiated.

The results of three MGP trials against each target are presented in Table 3. The numbers given in the parentheses are the total number of palpations, which is the sum of the eight presses for MGP and the number of local palpations. The expectation values of the number of trials needed to palpate all inclusions using only local palpations to carry out a global search, without the use of the MGP stiffness map, are 13 for Obj.1 and approximately 17 for Obj.2. In view of the expectation values for global searches and the results of Table 3, we can expect that global palpations using MGP reduce the number of local palpations. Stiffness maps obtained in experiment 3 (Exp.3) of Table 3 are shown in Fig. 9. The cells given higher priorities for local palpation are displayed in brighter shades of white while those with lower priorities are displayed with darker shades of gray approaching black. It can be seen that the cells with higher palpation priorities are not necessarily closer to the inclusions. This is because the stiffness maps display a degree of rotational symmetry due to the increased patterns based on rotation. A map displaying perfect rotational symmetry can be viewed as showing the priorities of the CC orbits. In such cases, we feel that the tradeoff curve between the number of palpations and search accuracy can be shifted to a higher performance range by carrying out MGP searches in half regions of the target or varying the number of convex shapes, before conducting local palpations.