1. Introduction

Robotic manipulators have evolved from performing predefined tasks in controlled settings to operating in dynamic and unstructured environments ^1,2.

Robotic grasping research has traditionally focused on model-based techniques that rely on accurate 3D representations of objects, environments, and hands ³. Although these methods can be effective in controlled environments, they often struggle to generalize to real-world scenarios because of their dependence on precise analytical models ⁴.

Despite these advancements, grasping capabilities remain a fundamental challenge, particularly in real-world environments where objects are often irregular, moving, or present in cluttered scenes. While robots are frequently designed for specific tasks in controlled settings ⁵, recent advances in robot learning have aimed at enhancing adaptability in unstructured scenarios ⁶. Robust and adaptive grasping is essential for robotic manipulation. However, identifying a stable and feasible grasp is non-trivial. This challenge is crucial because it underpins the ability of robots to autonomously perform complex tasks in dynamic and unstructured environments ⁷.

Vision-based approaches, including RGB-D sensors, have advanced robotic grasping ⁸. However, these systems often suffer from limitations such as occlusions, sensor noise, and difficulty in perceiving detailed contact information, particularly at close range and with transparent or reflective objects ⁹. In contrast, tactile sensing provides a fine-grained local geometry that is often unobservable through vision alone. Although tactile sensors also present challenges, such as limited spatial coverage and resolution, their ability to detect subtle physical interactions makes them especially valuable for evaluating and refining grasp quality in manipulation tasks ¹⁰.

Tactile sensing provides direct contact feedback, allowing real-time evaluation of grasp stability. By capturing physical interactions at contact points, tactile feedback has become essential for assessing grasp quality ^11,12. This sensory information is critical for converting physical interactions into controlled robotic actions, such as the regulation of gripping force ¹³.

Traditionally, grasp stability was assessed using deterministic approaches that analyze force resultants and contact mechanics ^14,15. Recently, data-driven models have predicted grasp stability based on tactile observations ¹⁶.

Despite advances in tactile sensing for grasp assessment, challenges remain. While current methods excel at evaluating grasp quality, limited guidance is provided for correcting unstable grasps. Although grasp-quality models predict stability, adaptive adjustments during manipulation have not been supported ¹⁷. Real-time tactile feedback is essential for correcting misalignments, insufficient force, or slippage; however, these methods may be inadequate when the initial grasp is unstable or when the system has low confidence in the prediction, underscoring the need for more reliable proactive grasp improvement techniques.

This highlights the need for proactive strategies, such as regrasping, which adjusts the grasp strategy in response to low confidence in the initial grasp. Regrasping leverages an assessment method to ensure stability and mitigate the predicted instability, thereby offering a more reliable solution for determining stability ^18,19. Previous research has not only focused on assessing the robotic grasp quality but also on improving it by predicting the unobserved contact area based on the current tactile image ²⁰.

By leveraging tactile data from the initial contact with the object, crucial information about the geometry of the object can be extracted to assess grasp stability. This study focuses on extrapolating the contact beyond the immediate contact area of the fingers, thereby enlarging the search space for candidate grasps.

At the CoRo Lab at École de Technologie Suérieure (ÉTS), prior research has advanced grasp assessment methods using tactile feedback, achieving high accuracy through unsupervised learning and CNN-based models. In this study, these methods were employed to assess grasp stability and to validate the extrapolated contact by comparing the tactile data extracted from the extrapolated contact with that of the real contact.

In contrast to approaches that rely on complex physical simulations, this study investigated the efficacy of geometric tactile extrapolation as a computationally efficient alternative. We present a framework in which a neural network (NN) classifies the shape of an object based on its initial tactile imprint. Based on this prediction, a shape-based completion procedure is used to infer the unobserved regions of the tactile map. This approach considers a tactile image as a local surface patch, enabling the identification of candidate grasps without immediate finger motion or full-object reconstruction.

It is important to note that the primary contribution of this work is the validation of this extrapolation signal as a reliable basis for future regrasping strategies, rather than the physical execution of the regrasp itself. This study evaluates the accuracy and reliability of the extrapolated contact using three key performance metrics: (1) the tactile centroid (TC) deviation, which measures the Euclidean distance between the geometric centers of the binary contact regions in the measured and extrapolated tactile maps; (2) the predicted grasp success rate (GSR), representing the statistical likelihood of a stable grasp as defined in ²¹; and (3) the structural similarity index (SSIM) to assess the local structural fidelity and perceptual consistency between the real and virtual tactile signatures. The focus of this work is to assess the effectiveness of the extrapolation step. Fig. 1 illustrates the high-level concept, in which the initial tactile image is used to infer unseen areas of contact.

This paper is structured as follows: Section 2 reviews the state-of-the-art in tactile regrasping and establishes the context for the proposed framework. Section 3 describes the experimental setup, NN classifier, contact extension method, and evaluation metrics. Section 4 presents the experimental results and discusses their implications. Section 5 provides a summary of the key findings and outlines directions for future research.

2. Contact Extrapolation as a Basis for Regrasping

Vision-based methods can struggle to accurately assess grasp and contact conditions, which are inherently tactile phenomena. Tactile sensing provides real-time information about key contact properties, such as contact location, geometry, and stability ²². Recent advancements have contributed to improving grasp robustness in dynamic and unstructured environments through tactile sensing ²³.

Grasp assessment can be performed using either model-based methods or machine-learning techniques. Model-based approaches grounded in grasp theory evaluate stability using parameters such as contact locations, surface normals, and force equilibrium. Wan and Howe ²⁴ developed a method to quantify the impact of tactile sensor spatial resolution on grasp stability, demonstrating that lower-resolution sensors introduce uncertainties in contact locations and surface normals, which, in turn, reduce the reliability of grasp stability predictions.

In contrast, machine-learning methods employ data-driven approaches to predict grasp stability, often enhancing performance through learned patterns and sensor fusion. Cockbum et al. ²⁵ developed an unsupervised learning method that achieved an 83.7% accuracy in detecting grasp failures. Kwiatkowski et al. ²⁶ later refined this approach by integrating exteroception and proprioception into a CNN-based model and achieved an accuracy of 92.7%. Recently, dataset augmentation techniques have further refined GSR estimation ²¹, highlighting the effectiveness of tactile sensing in predicting grasp stability and emphasizing its potential for intelligent feedback in robotic manipulations.

Similarly, Hyttinen et al. ²⁷ introduced a part-based approach to grasp stability, in which tactile signatures were learned from prototypical object parts. By combining tactile feedback with finger joint data and classifying tactile impressions, their model enhanced grasp stability predictions and facilitated dynamic adjustments during manipulation.

Beyond grasp assessment, the challenge lies in responding to predicted instability. This requires strategies that not only evaluate but also guide the system toward stable grasps. In human-robot collaboration, Tanaka et al. ²⁸ addressed this by using tactile sensors to detect interaction states, such as slipping, to dynamically adjust robotic grasping force.

Regrasping strategies in robotic manipulation often rely on visual and tactile feedback to improve grasp stability and adaptability. Vision-based methods, such as those proposed by Higo et al. ²⁹, utilize visual data to assess and adjust grasp configurations. Kalashnikov et al. ³⁰ enabled grasp adjustments before the final lift through closed-loop vision-based control, thereby facilitating behaviors such as probing and repositioning. By contrast, tactile-based methods are less commonly employed.

Hogan et al. ²⁰ proposed a regrasp control policy that improves the grasp quality using tactile feedback. A deep CNN trained in a self-supervised manner on more than 2,800 grasps provided a tactile-only grasp-quality metric. For each candidate local regrasp, they synthesized the post-move tactile imprint by applying rigid-body translations to the measured tactile image, filled the newly introduced regions via mirror padding, then scored these synthetic imprints with the learned metric, and executed the highest-ranked action. The procedure assumes that the object pose is unchanged between grasps, and emphasizes the local contact geometry rather than the object’s global shape. The reported results included 85% accuracy for known objects, 75% accuracy for novel objects, and a 70% average relative improvement in grasp success.

In contrast, the present work leverages tactile data as the basis for reconstructing the object geometry and identifying improved grasping positions, thereby providing a perceptual foundation for a shape-aware regrasp strategy. Instead of executing closed-loop regrasps, this study focused on validating the accuracy of the extrapolation mechanism. The framework virtually explores candidate adjustments by extrapolating the contact region and inferring unseen portions based on the object’s geometry. These virtual images were then evaluated using a grasp assessment network ²¹ to predict stability. By comparing these virtual signatures with real tactile data collected through systematic robot motion, we demonstrated that this extrapolation is sufficiently accurate to serve as a reliable signal for future regrasping controls.

The challenge of extrapolation based on different object geometries lies in accounting for variations in surface curvature. Spheres exhibit an isotropic, constant positive curvature (principal curvatures equal to \(1/R\)). Therefore, the completion of the tactile map can be symmetric around the contact normal. Cylinders have anisotropic curvature, are non-zero in the circumferential direction (approximately \(1/R\)), and are near zero along the axis; therefore, completion should be direction-dependent. Cuboids comprise piecewise planar faces (zero curvature) separated by sharp edges where curvature is concentrated; therefore, tactile-map completion must be edge-aware to avoid introducing fictitious curved contacts near the edges. The extrapolated tactile-map patterns varied across prototypical shapes (see Fig. 2(III)).

Inspired by Hyttinen et al.’s classifier-based grasp-stability predictions ²⁷, a shape-specific extrapolation strategy was introduced based on three prototypical shapes (cuboid, sphere, and cylinder) to enable structured, adaptive refinement. Although originally developed for a different purpose, this method has proven to be highly effective for mapping tactile images to prototypical shapes. Their approach involves reducing the graspable object space to a small set of representative shapes through data-driven clustering. By associating tactile feedback with specific parts of these shapes, their model improved the grasp stability predictions, enabling the robot to classify tactile signatures from different object regions. While applied in a different context, this work laid the foundation for the shape-specific regrasp strategy.

Assuming a continuous contact surface when extending tactile information is a common simplification, but it can fail to capture the complex and discontinuous contours of real-world objects. Instead, the proposed method employs a shape-specific extrapolation strategy, where object classification guides the extrapolation of the contact region. This explicitly differs from generic, shape-agnostic continuity extrapolation. Rather than extending the edges blindly, completion follows curvature-consistent rules tied to the estimated shape (sphere, cylinder, and cuboid). This enables more accurate extensions by aligning the extrapolated contact with the geometry of the object, rather than relying on generic continuity assumptions.

An NN classifier trained on tactile features ensures effective shape recognition and supports geometry-aware extrapolation. The resulting geometry-aware completion provides the basis for a proactive regrasp strategy that responds to the predicted instability by suggesting small, shape-informed adjustments in the grasp configuration.

3. Methodology

This section outlines the methodology for extrapolating unobserved contact regions by combining shape classification with tactile data. Fig. 2 summarizes the overall processing pipeline.

The experimental data were collected using a UR5e robotic arm equipped with a parallel-jaw gripper. Two multimodal capacitive tactile sensors ³¹, mounted on a Robotiq 2F-85 gripper, were used to capture the tactile images.

Developed and commercialized by the CoRo Lab, these sensors feature a \({7 \times 4}\) taxel array for normal pressure sensing at 60 Hz. Each sensor consists of 28 capacitive elements, an integrated inertial measurement unit with gyroscopes, accelerometers, and a compass for joint angle estimation relative to the palm of the gripper. A transimpedance amplifier captures capacitance variations at 1 kHz with a \({7 \times 4}\) taxel matrix representing static pressure data, whereas capacitance fluctuations and accelerometer readings provide dynamic information.

3.1. Shape Classifier Network

To extrapolate the contact beyond the initial imprint, the shape of the object was first classified from the initial tactile image. This classification, combined with the initial tactile image, formed the foundation for extrapolating the complete contact region.

The cuboid, sphere, and cylinder classes were selected as prototypical shapes representing a broad set of real-world objects ³². These shapes are widely used in robotics and object recognition for contact modeling. This choice supports the generalization of the contact extrapolation by covering a diverse range of contact geometries.

A fully connected NN was developed to classify object shapes based on tactile features, following a previous study ³³. The dataset comprised approximately 6,900 samples across three object classes, with each shape 3D-printed in five sizes. The object sizes were selected according to three criteria: (i) compatibility with the gripper stroke so that the object could be enclosed stably; (ii) tactile detectability such that the initial contact produced a contiguous patch on the sensing area; and (iii) for cylinders and, in some cases, cuboids, at least one dimension exceeding the sensor’s active length so that only a portion of the true contact patch was captured. For the cuboids, the dimensions along the gripper closing axis (thickness) were varied from 20 mm to 42 mm to promote a stable enclosure. Spheres had radii from 12.5 mm to 32.5 mm. Cylinders had diameters from 22 mm to 42 mm, and their lengths were set greater than the sensor’s active length.

The tactile data from both sensors were normalized and reshaped into a \({7 \times 8}\) matrix to preserve spatial structure and sensor-specific information. The dataset was divided into three subsets: 80% training, 10% validation (for monitoring and tuning during model development), and 10% strictly held out as the test set (i.e., previously unseen data). This test set was never used during the model development and provided a measure of the generalization ability of the model.

To enhance classification, three feature families were integrated: raw pixel data from the tactile imprints, Hu moments (seven invariants that capture global shape and are invariant to translation, rotation, and scale), and selected central moments (second- and third-order: \(\mu_{20}\), \(\mu_{02}\), \(\mu_{11}\), \(\mu_{03}\), \(\mu_{21}\), and \(\mu_{12}\)), which quantify the dispersion, orientation, and asymmetry of contact intensity about the centroid ^34,35. These moment-based features provide geometric/structural descriptors derived from the spatial distribution of the tactile contact, complementing the pixel representation and improving class separability. Together, these features enable more accurate shape classification.

The combined feature vector was normalized using MinMaxScaler and input into a feedforward NN comprising three hidden layers. The first layer contained 64 ReLU-activated neurons, followed by two 32-neuron ReLU layers with L2 regularization. A softmax output layer with three neurons corresponds to the shape classes. The model was compiled using the Adam optimizer (learning rate of 0.001) and categorical cross-entropy loss. Training was performed over 10 epochs with a batch size of 32. As shown in Fig. 3, the training and validation curves closely track each other, suggesting limited overfitting, and the final accuracies were 94.5% (training) and 94.3% (validation), with losses of 0.37 and 0.37, respectively.

To evaluate the generalizability of the model to real-world conditions, a separate external dataset was constructed using objects not previously seen in the network. Five novel objects were selected from each shape category. For cuboids, a range of widths was included to assess how edge configurations, especially narrow versus wide configurations, affected tactile imprints. Across all classes, the selected objects differed in size, shape, and surface texture, while maintaining similar rigidity. Each object was grasped at multiple positions and orientations, with the number of grasps determined by the object size and the fixed spatial step size of the robot. The objects included in the external validation are shown in Fig. 4.

The confusion matrices for the held-out test set and external real-world object set are shown in Fig. 5. The held-out test set (Fig. 5(a)) achieved 94.3% accuracy, whereas the external dataset (Fig. 5(b)) yielded an accuracy of 80.8%.

A notable source of misclassification is the coffee container (Fig. 4(c)), which is frequently identified as a cuboid. Its large diameter creates a locally flat tactile imprint, reducing the shape discriminability. This case accounted for 9 of the 14 misclassifications in the cylinder category.

3.2. Shape-Based Contact Extrapolation

Once the shape of the object has been classified, an extrapolation strategy is applied to extend the contact area based on both the shape category and initial tactile image. The following section outlines contact extrapolation methods tailored to the geometry of each object.

3.2.1. Contact Extrapolation

The original \({7 \times 4}\) taxel matrix was first smoothed using a Gaussian filter to reduce the noise and improve the contact accuracy. The filter was parameterized with standard deviations \(\sigma_{x}\) and \(\sigma_{y}\) controlled smoothing along the \(X\)- and \(Y\)-axes. The Gaussian function is

\begin{equation} G(x, y) = \frac{1}{2\pi \sigma_{x} \sigma_{y}} \exp \left(-\frac{x^{2}}{2\sigma_{x}^{2}} - \frac{y^{2}}{2\sigma_{y}^{2}}\right), \end{equation}

where \(G(x, y)\) represents the filter value at \((x, y)\), and \(\sigma_{x}\) and \(\sigma_{y}\) control the degree of smoothing in \(x\) and \(y\), respectively.

Next, the sensor matrix was padded on all sides to expand the tactile space with newly added taxel-assigned values based on the contact shape. Bicubic interpolation with an upsampling factor \(r = 10\) was then applied to generate a high-resolution tactile representation. This approach preserved the geometric integrity of different object shapes while ensuring continuity in the extrapolation of the contact area.

While prior research has quantitatively examined how sensor resolution affects grasp-stability prediction ²⁴, in this work bicubic interpolation was applied to enhance the spatial resolution of the tactile signal.

Bicubic interpolation extends the cubic interpolation to two dimensions using a third-degree polynomial to approximate a function based on known values and their derivatives. For example, if the function \(f(x)\) is known at \({x = 0}\) and \({x = 1}\) and if the derivatives of \(f(x)\) are known at these points, the function can be interpolated using a third-degree polynomial within the interval \([0, 1]\).

\begin{equation} p(x, y) = \sum_{i=0}^{3} \sum_{j=0}^{3} a_{ij} x^{i} y^{j} \label{eq:bicubic} \end{equation}

In Eq. \(\eqref{eq:bicubic}\), 16 coefficients \(a_{ij}\) are computed using the known values of the function \(f(x, y)\) and its derivatives.

Before applying any contact extrapolation strategy, the method verifies whether the contact footprint of the object is fully enclosed within the active area of the tactile sensor. If all the contact taxels lie within the sensor boundary without edge overlap, no extension is applied. This prevented unnecessary modifications and focused on cases with incomplete contact data. Consequently, for small objects fully contained within the sensing area, potential shape misclassifications do not affect the tactile map because shape-specific extension rules are not triggered for fully enclosed contact footprints.

3.2.2. Shape-Specific Rules

Each prototypical shape follows a tailored extension strategy.

Cuboid: For flat-surface objects, such as boxes and books, edge taxels are mirrored across the sensor boundary to extend the planar contact area, approximating a local rigid translation aligned with the sensor frame, consistent with the assumptions in ²⁰.

Sphere: For round, smooth objects (e.g., balls and knobs), when the spherical contact intersects a sensor edge and is not fully contained, newly added taxels are populated by symmetric reflection across that edge to continue the contact. This simplifying assumption preserves the spherical geometry exactly only when the intersection is near an equatorial cross-section (i.e., the centerline lies on the edge). Otherwise, it can over- or under-extend the curvature. It is adopted here as a baseline, and future work will estimate the local radius and center offset to enforce curvature-consistent completion.

Cylinder: The cylinder represents elongated objects, such as bottles and rods, combining elements of both cuboid and spherical shapes. When the contact intersects the sensor frame, it extends along its principal axis and symmetrically populates a region perpendicular to the axis.

In Fig. 6, representative initial and extrapolated contacts for each class are shown. The extended contact effectively enlarges the portion of the object represented in the sensor frame. This enlargement provides an extended contact surface, which can be leveraged in future studies to identify stable configurations. This promotes the spatial continuity of the tactile map and supports the prediction of grasp outcomes, thereby improving the reliability and generalizability of the contact extrapolation step.

3.3. Evaluation Metrics

After classifying the object and extrapolating the unseen contact region, quantitative evaluation metrics are used to assess the accuracy of the extrapolated contact. These metrics determine how closely the virtual data approximates the real tactile feedback.

Several studies have highlighted the importance of center-of-mass (CoM) estimation for adaptive grasping. For example, Wang et al. ³⁶ proposed a Bayesian framework to estimate the physical CoMs of unknown objects to aid recovery from failure. By contrast, the present study computes a TC from a tactile image and compares the centroids between the measured and extrapolated maps. The TC is the geometric centroid of the binary contact region (unweighted), which enables a direct comparison between real and virtual data.

3.3.1. Tactile Centroid (TC)—Area-Normalized MSE

The mean squared error (MSE) quantifies the alignment between the real and extrapolated contact positions. Each sensor is calculated as follows:

\begin{equation} \mathrm{MSE} = \frac{1}{n} \sum_{i=1}^{n} E_{i}^{2}, \end{equation}

where \(n\) is the number of evaluated grasp positions and \(E_{i}\) is the Euclidean distance [mm] between the real and virtual TC at position \(i\). Note that TC represents the local geometric center of contact within the \({7 \times 4}\) sensor frame, distinct from the object’s global center of gravity. For the rigid objects in this study, the mass distributions were relatively uniform, and the form factors were moderate, ensuring that the center of mass was not significantly offset from the grasp center. Thus, TC serves as a reliable local alignment indicator, allowing us to evaluate the extrapolation fidelity regardless of the global object dimensions. The area \(A\) is defined as

\begin{equation} A = P_{x} P_{y} R_{s}^{2}, \label{eq:area} \end{equation}

where \(P_{x}\) and \(P_{y}\) are the numbers of grasp positions explored along the \(X\)- and \(Y\)-axes (dimensionless), respectively, and \(R_{s}\) is the robot step size [mm]. Hence, \(A\) is in mm\(^{2}\). This area represents the total region covered during systematic data collection (not the physical sensor size or number of active taxels). To ensure a fair comparison across different objects and scenarios, the error was normalized by this area as follows:

\begin{equation} \mbox{Area-normalized~$\mathrm{MSE}_{\mathrm{S1}}$} = \frac{\mathrm{MSE}_{\mathrm{S1}}}{A}. \label{eq:mse_s1} \end{equation}

In Eq. \(\eqref{eq:mse_s1}\), the term \(\mathrm{MSE}_{\mathrm{S1}}\) represents the mean squared error associated specifically with Sensor 1 (\(\mathrm{S}1\)), which refers to the first of the two capacitive tactile sensor arrays mounted on the robotic gripper. A similar definition applies to Sensor 2. Because both \(\mathrm{MSE}\) and \(A\) are in mm\(^{2}\), the area-normalized MSE is dimensionless. Lower values indicate closer alignment between the real and virtual data.

3.3.2. Tactile Centroid (TC)—Absolute Error

The absolute error was computed by measuring the Euclidean distance between the extrapolated and actual TCs at each test point. Movements occurred along the \(X\)- and \(Y\)-axes. As the number of test points was limited by both the small object dimensions and the minimum step size of the robot, a grid-based sweeping procedure was employed. This method provides sufficient spatial coverage while capturing meaningful variations in tactile images (Fig. 1(e)).

The normalization and averaging procedures used to produce a unified TC error trend per object followed the same approach described for the GSR normalized relative error metric (see Section 3.3.4).

3.3.3. GSR—Area-Normalized Mean Absolute Error [% mm\(^{-2}\)]

The GSR provides a complementary metric to evaluate how closely the virtual tactile data aligns with real contact information. It is computed using the grasp assessment network introduced in ²¹, which employs an unsupervised feature-learning approach to estimate the likelihood of a successful grasp.

To quantify the extrapolation accuracy, the GSR values predicted from the extrapolated tactile images were compared with those obtained from real tactile data across all positions in the grid-based scan of the robot.

The mean absolute error (MAE), expressed in percentage points, is defined as

\begin{equation} \mathrm{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left|G_{i}^{\mathrm{real}} - G_{i}^{\mathrm{virt}}\right|, \end{equation}

where \(n\) is the total number of evaluated grasp positions (i.e., robot steps during the grid-based scan), and \(G^{\mathrm{real}}_{i}\) and \(G^{\mathrm{virt}}_{i}\) are the GSR scores computed for the real and virtual tactile images at position \(i\), respectively.

To allow fair comparisons across experiments with varying object sizes and grid resolutions, the MAE was normalized by the total area explored during the grid scan, yielding the area-normalized MAE:

\begin{equation} \mbox{Area-normalized $\mathrm{MAE}$} = \frac{\mathrm{MAE}}{A}, \end{equation}

where \(A\) is the total area defined in Eq. \(\eqref{eq:area}\). Since MAE is expressed in percentage points and \(A\) in mm\(^{2}\), the resulting area-normalized MAE has units of % mm\(^{-2}\). This metric represents the average percentage error per unit area, which enables a standardized evaluation across trials.

3.3.4. GSR—Normalized Relative Error

The normalized relative error quantifies the proportional discrepancy between the real and virtual GSR. It is defined as

\begin{equation} \mathrm{RelErr}_{i}~[{\%}] = 100 \dfrac{\bigl| G_{i}^{\mathrm{real}} - G_{i}^{\mathrm{virt}} \bigr|} {\max \bigl(G_{i}^{\mathrm{real}}, \varepsilon \bigr)}, \end{equation}

where \(G_{i}^{\mathrm{real}}\) and \(G_{i}^{\mathrm{virt}}\) are the real and virtual GSR values at grasp point \(i\), respectively, and \(\varepsilon>0\) is a small constant to avoid division by zero.

To evaluate how this error evolves across the grid, the values are arranged into a matrix \(E \in \mathbb{R}^{P_{x} \times P_{y}}\), where \(e_{i,j}\) is the relative error at grid point \((i,j)\) (row \(i\) indexes the \(X\)-step, column \(j\) indexes the \(Y\)-step):

\begin{equation*} E = \begin{bmatrix} e_{1,1} & e_{1,2} & \cdots & e_{1,P_y} \\ e_{2,1} & e_{2,2} & \cdots & e_{2,P_y} \\ \vdots & \vdots & \ddots & \vdots \\ e_{P_{x},1} & e_{P_{x},2} & \cdots & e_{P_{x},P_{y}} \end{bmatrix}. \end{equation*}

Each row in \(E\) corresponds to a sweep along the \(Y\)-axis starting from a different \(X\)-position. Minor variations occurred along \(X\) before moving farther from the initial contact edge along \(Y\). To normalize for differing error magnitudes at the initial positions, each row was divided by its first element. The first row represents the baseline sweep and was left unnormalized.

\begin{equation} e'_{i,j} = \left\{\begin{aligned} &e_{i,j}, &&i = 1, \\ &\dfrac{e_{i,j}}{\max \bigl(e_{i,1}, \varepsilon\bigr)}, &&i = 2, \dots, P_{x}, \end{aligned}\right. \quad j = 1, \dots, P_{y}. \end{equation}

For all normalized rows (\(i > 1\)), the first element equals one, \(e'_{i,1} = 1\), by construction. The row-normalized matrix is

\begin{equation*} E' = \begin{bmatrix} e_{1,1} & e_{1,2} & \cdots & e_{1,P_y} \\ e'_{2,1} & e'_{2,2} & \cdots & e'_{2,P_y} \\ \vdots & \vdots & \ddots & \vdots \\ e'_{P_x,1} & e'_{P_x,2} & \cdots & e'_{P_x,P_y} \end{bmatrix}. \end{equation*}

This row-wise normalization aligned the starting point of each sweep, enabling a comparison of the error evolution along the \(Y\)-axis. The normalized errors were then averaged across all \(X\) steps to obtain a single curve per object, as follows:

\begin{equation} \bar{e}_{j} = \frac{1}{P_{x}} \sum_{i=1}^{P_{x}} e'_{i,j}, \quad j = 1, \dots, P_{y}. \end{equation}

The resulting one-dimensional vector is

\begin{equation*} \bar{E} = \bigl[\bar{e}_{1}, \bar{e}_{2}, \dots, \bar{e}_{P_{y}}\bigr]. \end{equation*}

\(\bar{E}\) captures the average relative error at each depth step along \(Y\), smoothing out lateral fluctuations. It provides a standardized and compressed representation of the GSR error, supporting meaningful comparisons across objects and grid configurations.

4. Results and Discussion

This section evaluates the accuracy and effectiveness of the proposed method for extrapolating the unseen contact regions. Using tactile data from multiple objects, the framework extends the contact areas conditioned on the initial sensor readings and predicted object shape.

Tactile images were first captured from a starting grasp using two capacitive sensors ³¹. Based on this initial contact, the shape of the object was classified using the framework described in Section 3.1. A grid-based movement (Section 3.3.4), incorporating a sweep/normalization procedure, was then employed to scan the surface of the object and collect tactile data at multiple relative finger positions.

Six objects from each shape class were used for evaluation. A grid-based traversal area was defined for each object. The measured tactile data were collected by moving the robot fingers across the object surface, whereas the virtual tactile data were sampled point-by-point from the extrapolated tactile map derived from the initial imprint and shape label.

A total of 650 test points were collected across all the objects. A 3D-printed adapter (see Fig. 1(j)) held the objects upright, allowing the fingers to conform to their surfaces under a normal force without rigid anchoring.

The evaluation results obtained using the TC absolute error metric (Section 3.3.2) are shown in Fig. 7. This metric quantifies the difference between the TCs of real and virtual contacts at each test point. The fingers traversed each object in a grid pattern, typically 10 steps along the \(Y\)-axis and a variable number along the \(X\), depending on the object width. As shown in Fig. 1(e), the finger moved incrementally along \(Y\), returned to the edge, shifted by one step in \(X\), and repeated this until the entire grid was covered.

Each object produced a set of TC deviation curves across \(X\)-positions. These curves were row-normalized and averaged to obtain a single representative trend over 10 \(Y\)-steps, as described in Section 3.3.4. An example stack of TC deviation curves for a cuboid with a \({6 \times 10}\) grid (60 points) is shown in Fig. 7(a).

The combined curves in Fig. 7 do not start at zero because the initial \(X\)-step in each test corresponds to a real sensor contact, whereas the subsequent \(X\)-displacements are derived from extrapolated data. These small initial discrepancies lead to a non-zero starting value for the averaged trend. A similar evaluation procedure was applied to the GSR.

The results show a consistent increase in the absolute TC deviation with a positive \(Y\), i.e., as the finger moves away from the initial contact edge toward the interior of the object. Because only the boundary region is directly observed, predictions farther from that edge increasingly rely on shape-conditioned extrapolation, which increases uncertainty and error. This trend is consistent with the design assumption that the extrapolation accuracy degrades with the distance from the measured contact.

Unlike TC, which is geometric, the GSR metric was derived from the predicted grasp-success scores from the network proposed in ²¹. Fig. 8 shows the normalized relative GSR error across the six test objects per class, computed using the procedure outlined in Section 3.3.4.

Similar to the TC trends, the GSR error increases with displacement from the edge of the object, confirming that the prediction quality deteriorates with distance. This reinforces the reliability of the evaluation approach and highlights the natural limitations of contact extrapolation based solely on initial contact.

Since the grid size varies across tests due to differing object dimensions, two complementary metrics were introduced in Section 3.3 to enable fair cross-object comparisons. The area-normalized MSE, as described in Section 3.3.1, summarizes the overall alignment between the virtual and real contact regions across the entire grid. By normalizing the MSE with respect to the total scanned area, this metric compensates for grid size variations, allowing for a consistent evaluation of the contact quality, regardless of the object geometry.

Similarly, the area-normalized MAE, detailed in Section 3.3.3, quantifies GSR error in terms of % mm\(^{-2}\). This normalization ensures that GSR discrepancies are assessed independently of the grid density and object scale, making the metric suitable for evaluating both scalability and prediction accuracy.

As summarized in Tables 1–3, the area-normalized MSE for the TC ranges from 0.008 to 0.05 for cuboids, 0.003 to 0.10 for spheres, and 0.001 to 0.04 for cylinders. Correspondingly, the area-normalized MAE for GSR ranges from 0.005% to 0.10% for cuboids, 0.02% to 0.08% for spheres, and 0.01% to 0.10% for cylinders. The consistently low TC values indicate that the extrapolated contacts are geometrically close to the measured contacts, whereas the low GSR errors suggest that the extrapolated contacts preserve grasp-relevant cues, as assessed by the predictor of ²¹. Because GSR does not uniquely determine contact geometry, the performance is reported jointly: Section 4.1 defines a safe zone that requires both TC and GSR errors to be below thresholds.

4.1. Safe Zone

A safe zone was defined to identify the regions where the extrapolated contact aligned with the measured tactile feedback. It comprises grid locations where both the TC and GSR errors fall below the empirically derived thresholds, indicating agreement between the extrapolated and measured data.

Rather than setting arbitrary bounds, a data-driven approach was used, and the threshold was taken as the 70th percentile of the cumulative error distribution across all tests. This typically captures the majority of low-error points, while accommodating variations across shapes and grid sizes. Thus, the safe zone is the area in which both evaluation metrics (TC and GSR errors) are below the 70th percentile of their respective distributions. It should be emphasized that the safe zone represents regions of high extrapolation fidelity rather than high physical grasp stability.

A grid point is considered “safe” if the virtual and real metrics are aligned within the defined thresholds, thereby providing the robot with a reliable perceptual boundary. Within this area, the robot can trust the extrapolated contact map to inform its next movement, regardless of whether the predicted grasp success is high or low.

For this dataset, the corresponding thresholds were approximately 5 mm (TC) and 34% (GSR). These percentile thresholds were derived from an analysis of the error distributions across all 18 objects and 650 test points, as illustrated in Fig. 9, which presents histograms of the TC and GSR errors along with their 70th percentile markers.

To quantify the extent of the safe zone relative to the total scanned grid, we compute

\begin{equation} \mbox{Safe Zone [{\%}]} = 100 \frac{S_{a}}{A}, \end{equation}

where \(A\) is the total grid area (see Eq. \(\eqref{eq:area}\)), and \(S_{a}\) is the cumulative area of all grid cells that meet both error criteria. This is calculated as

\begin{equation} S_{a} = N_{s} A_{c}. \end{equation}

Here, \(N_{s}\) denotes the number of qualifying grid points, and \(A_{c}\) is the area of a single cell, defined by the robot’s movement step size. Each step spans a rectangular region, and \(A_{c}\) is computed accordingly. For each object category, a representative sample was selected to illustrate a safe zone. These examples demonstrate the spatial extent to which both the TC and GSR errors remain below their respective 70th-percentile thresholds, indicating reliable extrapolated contact. Fig. 10 visualizes these zones, with hatched black outlines highlighting the regions of reliable alignment between the virtual and real contacts.

A full summary of the safe zone percentages across all objects is provided in Table 4, accounting for the varying grid sizes across object geometries.

4.2. Supplementary Metric: SSIM Analysis

The SSIM ³⁷ was computed to further evaluate the fidelity of the extrapolated contacts. SSIM is widely used for assessing perceptual similarity and structural consistency between images and has also been applied to tactile sensing ³⁸. Whereas the TC error captures the geometric alignment, the SSIM reflects the preservation of the local structure, providing a complementary perspective.

Figure 11 shows the relationship between the TC deviation and SSIM for all test points and sensors. The vertical and horizontal dashed lines mark the median TC deviation and SSIM, respectively, partitioning the scattering into four quadrants. A moderate negative correlation (Pearson’s \(r = -0.64\)) was observed; lower TC error tended to coincide with higher SSIM. The marginal densities further indicate that many test points fall in the quadrant with a low TC error and high SSIM under the present protocol.

Taken together, these analyses suggest that the extrapolated contacts not only align geometrically with the measured contacts (TC) but also preserve the local structure (SSIM).

5. Conclusion

This paper presented a tactile-based framework for extrapolating unseen contact regions to support robotic regrasping. A shape classifier assigned objects as cuboids, spheres, or cylinders based on the initial tactile imprint, after which the unobserved portions of the tactile map are extrapolated using shape-specific rules. Accuracy was assessed using multiple metrics, including the TC, GSR, and SSIM, and further summarized with a safe zone criterion that combines TC and GSR.

In the held-out test set, the shape classifier reached 94.3%; on an external set of unseen real-world objects, 80.8%. Across objects, area-normalized TC MSE (dimensionless) ranged from 0.008 to 0.05 for cuboids, 0.003 to 0.10 for spheres, and 0.001 to 0.04 for cylinders, while area-normalized GSR MAE ranged from 0.005% mm\(^{-2}\) to 0.10% mm\(^{-2}\), 0.02% mm\(^{-2}\) to 0.08% mm\(^{-2}\), and 0.01% mm\(^{-2}\) to 0.10% mm\(^{-2}\), respectively (Tables 1–3). SSIM analysis showed a moderate negative correlation with TC error (Pearson’s \(r = -0.64\)), indicating that lower centroid deviation tended to coincide with higher structural similarity. A recurring failure mode involved large-radius cylinders (e.g., the coffee container), whose locally flat imprints led to misclassification as cuboids (9 of 14 cylinder misclassifications), which is consistent with the limited curvature cues in the sensed contact.

Although effective under the tested conditions, this method could also benefit from improved adaptability. The shape classifier performed well on small, rigid objects, but struggled with larger ones. Expanding the dataset to encompass a wider range of shapes and scales could improve classification and enhance contact extrapolation. Similarly, refining the grasp assessment network may yield more reliable GSR predictions and support more accurate safe zone identification.

Incorporating higher-resolution tactile sensors, as suggested by prior studies on resolution effects ²⁴, could improve the extrapolation method by capturing finer contact details, potentially leading to more precise contact estimates and more stable grasp predictions. In combination with broader training data and an updated assessment network, these changes may increase overall accuracy and practical utility.

Future studies should leverage the validated extrapolation signal to implement a virtual search strategy for closed-loop regrasping. The objective was to use the extrapolated map to identify stable grasp configurations near the initial contact, limiting the physical finger reconfiguration and compensating for misalignments. This proactive approach aims to reduce failure risks such as tipping or slippage before the final lift.

Although the TC was computed from binary tactile images for simplicity, future implementations could leverage full-valued tactile intensities. This richer input may enable a more accurate estimation of the contact distribution and better reflect the force variations across the contact region.