Hand closure model for planning top grasps with soft robotic hands

Abstract

Automating the act of grasping is one of the most compelling challenges in robotics. In recent times, a major trend has gained the attention of the robotic grasping community: soft manipulation. Along with the design of intrinsically soft robotic hands, it is important to devise grasp planning strategies that can take into account the hand characteristics, but are general enough to be applied to different robotic systems. In this article, we investigate how to perform top grasps with soft hands according to a model-based approach, using both power and precision grasps. The so-called closure signature (CS) is used to model closure motions of soft hands by associating to them a preferred grasping direction. This direction can be aligned to a suitable direction over the object to achieve successful top grasps. The CS-alignment is here combined with a recently developed AI-driven grasp planner for rigid grippers that is adjusted and used to retrieve an estimate of the optimal grasp to be performed on the object. The resulting grasp planner is tested with multiple experimental trials with two different robotic hands. A wide set of objects with different shapes was grasped successfully.

Keywords

Grasping grasp planning soft robotic hands hand closure model

1. Introduction

Autonomous grasping in unstructured environments still remains an open challenge in robotics (Bohg et al., 2014), as it requires the development of advanced software (algorithms for grasp planning and control) and hardware (robotic hands and grippers). As robots get closer to humans, their ability to safely interact with unstructured environments proves to be much more important than their force or velocity performance, and the adoption of soft end-effectors becomes fundamental (Dollar and Howe, 2010). Compliant and underactuated robotic hands are mechanisms that have been studied since the late 1970s (Birglen et al., 2008; Hirose and Umetani, 1978). More recently, advancements in the field of soft robotics boosted the creation of the first completely soft manipulators and grippers (Hughes et al., 2016).

Along with the design of novel devices, new ways of implementing grasp planning and control strategies must be developed. In this respect, research on soft manipulation has significantly changed the way roboticists look at the problem of grasping. Thanks to the presence of passive compliance in their mechanical structures, soft hands are adaptable, versatile, and robust. Thus, part of the “intelligence” of the grasping system can be embodied in the hardware design, instead of being allocated exclusively to the planning and control algorithms. This is one of the main reasons why classic grasp planning methods, prescribing the exact position of fingers for ensuring force closure, need to be extended, or even overcome, so to exploit the intrinsic features of soft hands (Brock et al., 2016).

To effectively exploit the softness of end-effectors and achieve robust grasps, features of the environment such as flat surfaces, walls, edges, and others, can be used as a help, implementing strategies similar to those adopted by humans (Eppner et al., 2015). Environmental constraints exploitation (ECE) with soft hands opens up possibilities that were inconceivable with rigid grippers (Bimbo et al., 2019; Brock et al., 2016; Hang et al., 2019).

This work focuses on grasp planning strategies to let soft robotic hands perform power and precision grasps from the top (top grasps), possibly exploiting the surface of the table to increase the grasp robustness when grasping small objects. Eppner et al. (2015) defined this type of grasp as surface-constrained.

The main contributions of this article are (i) laying the theoretical foundations and (ii) proposing a practical implementation of grasp planning strategies for performing top grasps with soft robotic hands. The proposed grasp planner relies on a method for modeling hand closure, the closure signature (CS), whose computation was first introduced by Pozzi et al. (2018) and is detailed and analyzed extensively in this article, completing and extending our previous work to different robotic hands and different closures. The CS maps a given hand closing motion to the direction along which the largest deformation occurs. This direction is akin to the grasping direction of a parallel-jaw gripper. Based on these considerations, in this work, we show that it is possible to plan top grasps by aligning the CS of the robotic hand to a suitable direction on the object, computed using a grasp planner for parallel-jaw grippers. In particular, we integrated the CS-alignment (see Figure 1) with a state-of-the-art grasp planner for parallel-jaw grippers based on deep learning (Mahler et al., 2017). Experiments conducted with two soft hands, the Pisa/IIT SoftHand (Catalano et al., 2014) and the RBO Hand 2 (Deimel and Brock, 2016), attached to a collaborative robot arm, showed that this integration is feasible and that different CSs can be used to perform different grasps (power/precision).

Fig. 1.

Block scheme of the CS-alignment proposed in this article. Given a certain robotic hand and a possible closing motion that it can perform, the CS indicates a “preferred grasping direction” for that closure, that can be thought as the grasping direction of a parallel-jaw gripper and can be aligned with a suitable direction on the object.

2. Related work

Planning grasps with soft hands needs a radical shift from classic grasp planners designed for rigid hands. The underlying principle of “rigid” planners was to identify a set of possible contact locations on the object and search for possible hand configurations able to match goal contact areas with the actual contact points (Borst et al., 1999; Miller et al., 2003). In several planners, possible achievable grasps were evaluated in simulation and sorted according to a quality measure (Roa and Suarez, 2015). In this process, the environment was always considered as a disturbance and so any contact was avoided. Soft hands, instead, allow the exploitation of environmental constraints to reduce the uncertainty on object location and can easily adapt to different object shapes thanks to their intrinsic compliance. Grasp planners developed for fully actuated hands can hardly take advantage of the soft hands features and often cannot be used since they require hand models (Prattichizzo et al., 2013a) and grasp quality measures (Haas-Heger et al., 2018; Pozzi et al., 2017), which are still open research problems for soft underactuated hands. First attempts to exploit soft hands features in planners (Bonilla et al., 2015; Eppner et al., 2015) mostly focused on object and environment perception, and used simple strategies once the object/environment was parsed correctly. The main idea was that the hand can then autonomously cope with different object shapes.

In this work, we show that, by using the information encapsulated in the CS, we improve the number of successful grasps. We also demonstrate that the CS can also be exploited to achieve precision grasps. In the approach of Pozzi et al. (2018) the objects were represented by their bounding box (Barequet and Har-Peled, 2001). To cope with objects that can hardly be approximated by a single bounding box (e.g., hollow boxes, bowls), more advanced techniques can be used, e.g., bounding box object decomposition (Bonilla et al., 2015; Huebner et al., 2008). However, these techniques usually come with a high computational burden, and are not suitable for on-line grasp planning. In this work, we decided to use a grasp planner based on a convolutional neural network (CNN) that is not only very efficient, but also shows high success rates and advanced generalization capabilities over new objects (Mahler et al., 2017). This grasp planner outputs an estimate of the optimal grasp to be performed given an RGB-D image of the object. One of the contributions of this article is to show that such a grasp planner, designed for parallel-jaw grippers, can also generate successful grasps for more complex hands by applying the hand CS representation.

In the last decade, several works in which an artificial neural network provides the relevant information needed to command a robotic end-effector to perform successful grasps have been published. The input of the network is usually an RGB or RGB-D image of the object to be grasped. Earlier works that exploited this synergy between robotic grasping and artificial intelligence obtained only the grasp position (i.e., the cartesian coordinates) from the employed neural network. In this respect, Saxena et al. (2008) were the first to propose an algorithm for grasping novel objects when an object 3D model is not available. They collected synthetic training data and developed a machine able to predict grasping points from 2D images of the objects to be grasped. Recent works introduced deep learning for finding the optimal grasp given an RGB-D image of the object (Lenz et al., 2015). Physical grasp trials were performed by Levine et al. (2018) and Pinto and Gupta (2016) to collect training data for their networks. Some of the latest contributions in the field of deep learning applied to robotic grasping come from the Dex-Net project.¹ Starting from the Dex-Net 1.0 dataset (Mahler et al., 2016), grasp planners for parallel-jaw grippers (Mahler et al., 2017) and suction cups (Mahler et al., 2018) have been developed. Recently, also a work presenting “ambidextrous” robot grasping has been published (Mahler et al., 2019).

Despite their advantageous features, underactuated and compliant robotic hands also present open challenges (Brock et al., 2016; Catalano et al., 2014; Deimel and Brock, 2016; Dollar and Howe, 2010). Designing them is difficult, as well as predicting, modeling, and controlling their function. Recent research in this field focused on finding answers to the growing need of mathematical models and shared standards for dealing with soft manipulation. Grasp quality indexes for underactuated and compliant grasps have been studied in Pozzi et al. (2017), and the quasi-static model of the grasp was extended to include hand–environment and object–environment interactions in Salvietti et al. (2015). Other works focused on studying the exploitation of environmental constraints to achieve robust grasps (Eppner et al., 2015), whereas the model presented in Pozzi et al. (2018), answers to the need for simple and functional ways to model multi-fingered robotic hands.

To the best of the authors’ knowledge, there are relatively few works combining AI-driven robotic grasping and manipulation with soft underactuated hands. Gupta et al. (2016) combined learning by demonstration and reinforcement learning techniques to achieve different manipulation tasks with an RBO Hand 2. In Wang and Ling (2016), instead, a geometric model of a three-fingered soft hand is exploited to perform power grasps. Given its particular structure, the hand is modeled as a cube, and the point cloud representing the object to be grasped should be contained in the cube; the grasp to be executed is returned by a CNN. A different approach was adopted by Choi et al. (2018), where a CNN outputs the approaching direction and orientation for grasping objects. The proposed network is trained and tested with grasps performed with a four-fingered soft gripper. In that work, hand softness is not modeled explicitly, but is implicitly encapsulated in the CNN training set, which was retrieved by trial and error. The use of this procedure leads to hand-specific grasping strategies, strongly reducing the generalizability to different robotic set-ups.

3. Functional modeling of hand closures

Pozzi et al. (2018) introduced the closure signature (CS) as a model to describe a closure motion of a robotic hand. The CS model aims at capturing the hand function, rather than representing its kinematic structure. This results particularly suitable for the description of soft hands.

The CS is specific for a given hand and a given closing motion performed by the hand. It is a way to represent how the hand closes in free space, i.e., without considering interaction with a grasped object or an external surface. This choice has been made because we are interested in modeling the hand capabilities in the general case, without the definition of a specific set of graspable objects.

A set of reference points must be defined on the robotic hand to compute the CS. The position of the reference points changes according to the hand configuration during the closure motion to be modeled. Consider, for instance, a set of reference points defined on the fingertips of an anthropomorphic hand model. When fingers flex to perform a power grasp starting from the hand completely open, the position of reference points changes as they reach their final positions close to hand palm. Such position transformations can be represented as a 6D rigid displacement and a non-rigid deformation of the cloud of reference points (Salvietti et al., 2014).

The CS of a robot hand performing a closure motion is defined as the direction along which the set of reference points is most deformed and is characterized by an application point $o_{h}$ and a unit vector $v_{CS}$ .

In principle, the CS could be applied to any closing motion of any robotic hand. However, it is mainly conceived for soft hands for two main reasons. First, rigid hands can be modeled as kinematic chains using well-known kinematics and dynamics principles. Second, intrinsically soft hands can deal with uncertainties by self-adapting to possible collisions with the environment or with the object to grasp. This allows us to take advantage from a simple model, the CS, of a complex system to do grasp planning. In addition, soft hands are difficult to model with a unique formalism, as they have very different designs and actuation systems. The CS was introduced to fill this gap.

3.1. Computation of the CS

The computation of the CS requires the definition and tracking of a set of reference points on the hand. Motions of the reference points are then represented by the computation of an equivalent homogeneous transformation, indicated with $T$ in the following.

For a given robotic hand, let us consider $n_{ref}$ reference points. To define the CS we have to track the reference points during the analyzed closure motion of the hand in free space. At the end of the tracking procedure we have a path with $n_{s}$ samples for each reference point

r_{j} (k) \in R^{3}, j = 1, \dots, n_{ref}, k = 1, \dots, n_{s}

The method to compute the CS can be applied to different choices of reference point locations, as long as they suitably describe the hand’s motion during closure. As an example, if during a hand closure the palm is still, reference points should be placed in the fingers rather than in the palm. In Gioioso et al. (2013), it was demonstrated the low sensitivity to reference point positions for this representation of transformation. The rationale for reference points locations is to favor the parts of the hand that move more during the selected closing motion.

Let us indicate with $r_{j, in} = r_{j} (1)$ and $r_{j, fin} = r_{j} (n_{s})$ the coordinates of a generic reference point in its initial and final configuration, respectively. We assume that the configuration variation of the reference points can be represented as a linear transformation, i.e., for each point $r_{j} \in R^{3}$ , $j = 1, \dots, n_{ref}$ , the following linear relationship can be written

{\hat{r}}_{j, fin} = T {\hat{r}}_{j, in}

(1)

where the symbol ^ indicates the homogeneous representation of the vector. The $4 \times 4$ matrix

T = [\begin{matrix} t_{1} & t_{2} & t_{3} & t_{4} \\ t_{5} & t_{6} & t_{7} & t_{8} \\ t_{9} & t_{10} & t_{11} & t_{12} \\ 0 & 0 & 0 & 1 \end{matrix}] = [\begin{matrix} A & b \\ 0 & 0 & 0 & 1 \end{matrix}]

(2)

is a homogeneous matrix, where $A$ is a $3 \times 3$ matrix representing the linear map and $b$ is a 3D vector representing the translation in the transformation (Siciliano et al., 2009).

Let us define the vectors $r_{in} = [r_{1, in}^{T}, \dots, r_{n_{ref}, in}^{T}]^{T}$ and $r_{fin} = [r_{1, fin}^{T}, \dots, r_{n_{ref}, fin}^{T}]^{T}$ containing the initial and final positions of all reference points. According to Equation (1), we can write

{\hat{\hat{r}}}_{fin} = [\begin{matrix} T & 0 & \dots & 0 \\ 0 & T & \dots & 0 \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & \dots & 0 & T \end{matrix}] {\hat{\hat{r}}}_{in}

where the symbol $\hat{^}$ indicates that the vectors contain the homogeneous representations of the single vectors they include. To find matrix $T$ we can solve the linear system

r_{fin} = Mt

(3)

where matrix $M \in R^{3 n_{ref} \times 12}$ is defined as

M = [\begin{matrix} M_{1} \\ \dots \\ M_{n_{ref}} \end{matrix}], M_{j} = [\begin{matrix} {\hat{r}}_{j, in}^{T} & 0_{1, 4} & 0_{1, 4} \\ 0_{1, 4} & {\hat{r}}_{j, in}^{T} & 0_{1, 4} \\ 0_{1, 4} & 0_{1, 4} & {\hat{r}}_{j, in}^{T} \end{matrix}] \in R^{3 \times 12}

and vector $t \in R^{12}$ contains the components of $T$

t = {[\begin{matrix} t_{1}, \dots, t_{12} \end{matrix}]}^{T}

The resulting $T$ matrix represents the average transformation between the initial and final reference points positions.

The solution of the system depends on the properties of matrix $M$ . We discuss each case in the following. Applications described in this article fall into cases 2 and 3.

Case 1: if the rank of matrix $M$ is lower than 12, the system admits infinite solutions, in other terms, it is not possible to identify a unique $T$ matrix. This problem occurs when the number of reference points is lower than four, or if in their initial configuration they are on the same plane. If it is not possible to solve this issue, for example in simple planar grippers, we could reformulate the problem by considering more restrictive conditions on $T$ elements. For example, if a planar transformation is assumed, only six elements of $T$ are independent.

Case 2: if matrix $M$ is square and full-rank, the system has a unique solution that can be defined by evaluating $M$ inverse. In this case, $T$ matrix corresponding to the evaluated $t$ , exactly satisfies Equation (1) for all the reference points (apart from numerical approximations necessary to evaluate the inverse of $M$ ).

Case 3: if matrix $M$ is full-rank, but $n_{ref} > 4$ , the system does not admit a solution in general, i.e., it is impossible to find a homogeneous transformation able to represent the displacement of all the reference points. In this case $T$ matrix elements evaluated by means of $M$ pseudo-inverse are those minimizing the overall quadratic error, i.e.,

∥ Mt - r_{fin} ∥^{2} = {| | [\begin{matrix} T & 0 & \dots & 0 \\ 0 & T & \dots & 0 \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & \dots & 0 & T \end{matrix}] r_{in} - r_{fin} | |}^{2}

Then, in general, if we apply the average transformation $T$ that we evaluated through the pseudo-inverse to each single reference point, we obtain

{\hat{r}}_{j, fin} \neq T {\hat{r}}_{j, in}

In this case, transformation $T$ represents an average approximation of all the motions of the reference points. Here we rely on the use of homogeneous matrices and on the solution of a simple linear system, because we have small displacements and we are interested in an overall description of the motion, not in the perfect reconstruction between the two sets of points. More accurate methods to retrieve such transformation could rely on strategies that, given two sets of points, recover the correspondence and the mapping between one set and the other, and are typically used in computer vision (Myronenko and Song, 2010).

Matrix $T$ represents a generic linear transformation, that can be further decomposed into simpler transformations. In the literature there are different procedures to decompose a generic $4 \times 4$ matrix into a composition of primitive transformations (Goldman, 1991). With a complete analysis of the $T$ matrix we can extract each single transformation component. For example, methods are available in Goldman (1991) for writing $T$ as the product of the following elements

T = T_{trans} T_{rot} T_{sh, xy} T_{sh, xz} T_{sh, yz} T_{scale}

The element $T_{trans}$ represents translation component and is simply given by

T_{trans} = [\begin{matrix} I_{3 \times 3} & b \\ 0 & 0 & 0 & 1 \end{matrix}]

The element $T_{rot}$ represents a rigid rotation matrix and is defined as

T_{rot} = [\begin{matrix} R & 0 \\ 0 & 0 & 0 & 1 \end{matrix}]

where $R$ is a $3 \times 3$ matrix such that $R R^{T} = I_{3 \times 3}$ , and $det (R) = 1$ (or, in other terms, $R \in SO (3)$ ).

The elements $T_{sh, xy}$ , $T_{sh, xz}$ , and $T_{sh, yx}$ represent three shear deformations; $T_{sh, xy}$ has the following structure

T_{sh, xy} = [\begin{matrix} 1 & s_{xy} & 0 & 0 \\ s_{xy} & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix}]

where $s_{xy}$ represents shear deformation along the plane $xy$ , $T_{sh, xz}$ and $T_{sh, yz}$ are similarly defined.

The element $T_{scale}$ represents a scale deformation and its structure is

T_{scale} = [\begin{matrix} s_{x} & 0 & 0 & 0 \\ 0 & s_{y} & 0 & 0 \\ 0 & 0 & s_{z} & 0 \\ 0 & 0 & 0 & 1 \end{matrix}]

where $s_{x}$ , $s_{y}$ , and $s_{z}$ represent scale factors along axis $x$ , $y$ , and $z$ , respectively.

In this work, we are mostly interested in decomposing the overall transformation into a rigid-body motion $T_{rb}$ and a deformation $T_{def}$

T_{rb} = T_{trans} T_{rot}, T_{def} = T_{sh, xy} T_{sh, xz} T_{sh, yz} T_{scale}

The transformation matrix $T$ can be expressed as

T = T_{rb} T_{def}

We relate the CS to $T_{def}$ assuming that the non-rigid part of the motion is strictly related to the grasp tightness. This consideration comes from literature on grasp analysis. Prattichizzo et al. (2013b) demonstrated that, in a grasp such that internal forces can be exerted, i.e., the object can be squeezed, the rigid-body motions are defined as coordinated hand/object motions in which contact forces are constant, while the non-rigid deformation is related to contact force variations. Bicchi (1995) showed that contact force variations due to hand activation are internal, i.e., they are self-balanced, and demonstrated that they play a key role in grasp stability. In other words, the rigid-body component mainly contributes to the motion of the grasped object, while the non-rigid part is strictly related to the grasp tightness. This distinction characterizes the hand motion itself, and it is not object-dependent. This is why we apply it to the CS computation, even though it is based on the analysis of the free hand motion. Among the available methods in literature to decompose a generic $4 \times 4$ matrix into a product of simpler transformations, we require one that allows the distinction between rigid and non-rigid components. The extraction of the translation part of the rigid-body motion from the starting matrix $T$ is straightforward considering (2). The matrix $A$ in Equation (2) can be written, with the polar decomposition, as the product

A = RU

in which $R \in SO (3)$ is orthogonal and represents a rigid rotation, and $U$ is a positive-semidefinite Hermitian matrix that takes into account the non-rigid deformation (Higham and Schreiber, 1990). $U$ in general contains both anisotropic scaling and shear deformation components, however it is possible to identify three principal directions defining a frame with respect to which the transformation is a pure anisotropic scaling. Such directions can be identified as the eigenvectors of $U$ matrix, while the corresponding eigenvalues represent scaling factors.

Definition 1 (Closure signature). Let us indicate with $λ_{i}$ and $v_{i}$ ( $i = 1, 2, 3$ ) the eigenvalues and the eigenvectors of the matrix $U$ , respectively, and with $o_{h}$ the centroid of the reference points trajectories computed as $o_{h} = mean (r_{j} (k) \in R^{3}$ , $k = 1, \dots, n_{s}$ , $j = 1, \dots, n_{ref})$ . The CS is defined as the direction of the eigenvector corresponding to the smallest eigenvalue of $U$ , $v_{CS} = [v_{cs, x}, v_{cs, y}, v_{cs, z}]^{T}$ , applied in $o_{h}$ .

Underactuated and compliant hands tend to perform power grasps (Pozzi et al., 2017), and this is why we initially focused on computing the CS of complete closures (Pozzi et al., 2018). The CS, however, can also be used to describe other types of closures, including other power grasps (e.g., lateral pinch defined by Cutkosky (1989)) and precision grasps.

3.2. Simplified example

To better describe all the steps that lead to the computation of the CS here we present a simple example with a set of six reference points initially placed in

r_{in} = [\begin{matrix} 1 & - 1 & 0 & 0 & 0 & 0.0001 \\ 0 & 0 & 1 & - 1 & 0 & 0 \\ 0.001 & 0.001 & 0 & 0.0001 & 1 & - 1 \end{matrix}]

If we apply a rigid rotation of $Rotz (π / 2)$ to them, we obtain a transformation $T$ with $A = RU$ , and $R = Rotz (π / 2)$ and $U = I_{3 \times 3}$ . Since there is no deformation, the CS cannot be computed (Figure 2a).

Fig. 2.

Example of CS computation for different points transformations. (a) Rotation of $π / 2$ about the $z$ -axis. (b) Deformation along the $x$ -axis. (c) Rotation and deformation. Initial points are indicated with red dots and numbers and the related final points are shown with black asterisks and numbers. The blue vector represents the computed $v_{CS}$ .

Applying only a deformation along the $x$ -axis to the first two points as in Figure 2b, we obtain the final position:

r_{fin} = [\begin{matrix} 0.1 & - 0.1 & 0 & 0 & 0 & 0.0001 \\ 0 & 0 & 1 & - 1 & 0 & 0 \\ 0.001 & 0.001 & 0 & 0.0001 & 1 & - 1 \end{matrix}]

Then, we can compute $T$ and its decomposition that results in having $R = I$ and

U = [\begin{matrix} 0.1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix}]

(4)

The smallest eigenvalue of $U$ , that is $0.1$ , corresponds to the scaling value of the applied deformation. The direction of the related CS is $v_{CS} = [1, 0, 0]$ , that is along the $x$ -axis, as expected. In case both previous transformations are applied (Figure 2c), the rigid rotation is $R = Rotz (π / 2)$ and $U$ is equal to (4). Thus, the CS direction is still $v_{CS} = [1, 0, 0]$ .

3.3. CS for power grasps

The CS of the Pisa/IIT SoftHand complete closing motion (i.e., from completely open to completely closed) was computed in Pozzi et al. (2018). The top grasps performed using this information had greater success rate and robustness with respect to a baseline method.

In this work, the generality of the CS computation is shown by applying it to another soft hand, the pneumatically actuated RBO Hand 2 (Deimel and Brock, 2016), that is significantly different from the Pisa/IIT SoftHand.

For the RBO Hand 2, a complete closure of all the fingers resembling the first hand synergy (see Salvietti, 2018) is considered. It is obtained by actuating all the hand degrees of actuation. Seven reference points, five on the fingertips and two on the palm, are chosen and tracked with a motion capture system (Motion Analysis). Figure 3a shows the initial and final position of the RBO Hand 2 during the tracking procedure from two different view points.

Fig. 3.

Power grasps with the RBO Hand 2. (a) Side (left) and top (right) views of the “heavy wrap” closing motion: from fully open to fully closed hand. Reference points are indicated with red dots over the open hand. (b) Example of “lateral pinch”.

From the same set of retrieved data we analyzed two different hand motions: one considering all the seven reference points and corresponding to a “heavy wrap” (Cutkosky and Kao, 1989), and one considering a subset of four reference points and corresponding to a “lateral pinch” (see Figure 3b). Figure 4 reports initial and final positions of the reference points and CS corresponding to these two different motions. Note that the second only considers the motion of the two points on the palm and of the fingertips of the thumb and index. In a reference frame placed on the hand wrist, with $x$ -axis pointing towards the middle finger, $y$ -axis pointing in the direction of the thumb, and $z$ -axis exiting from the plane, in the first case we have that $v_{CS} = [0.92, - 0.06, - 0.39]$ mm and $o_{h} = [100, 5, - 30]$ mm, and in the second case $v_{CS} = [0.88, 0.27, - 0.38] mm$ and $o_{h} = [89, 29, 35]$ mm. The CS direction, $v_{CS}$ , slightly varies between the two cases, but this is not the most relevant aspect. What really makes the difference in the closure description and also in the experiments, is that in the second case the center of the CS, $o_{h}$ , falls between the thumb and the index, thus allowing a good alignment with the object to perform a lateral pinch, as in Figure 3b.

Fig. 4.

RBO Hand 2: CS computation ( $o_{h}$ , $v_{CS}$ ) for two different closing motions. The hand is a right hand and is seen from the top. The points are expressed in a reference frame placed on the wrist, with $x$ -axis pointing towards the middle finger, $y$ -axis pointing in the direction of the thumb, and $z$ -axis exiting from the plane. Magenta circles and red stars indicate initial and final positions of the reference points, respectively. (a) CS of the heavy wrap motion (from the fully open hand to the fully closed hand, as in Figure 3a), when considering seven reference points: five in the fingertips, two in the palm. (b) CS of the lateral pinch motion that corresponds to that shown in Figure 3a, when actuating only index, palm, and thumb. The resulting grasp is shown in Figure 3b. Reference points are placed in the palm and in the fingertips of the thumb and of the index.

In both closures, taking the palm into account is fundamental, as it is actuated during the considered grasp, and it allows the hand to wrap objects. Without considering the two palm points in the heavy wrap, for example, the CS changes to $v_{CS} = [0.65, - 0.67, - 0.36] mm$ and $o_{h} = [111, 6, - 35] mm$ . As shown in Figure 5, the resulting CS is more in the direction of the thumb motion than in Figure 4a.

Fig. 5.

RBO Hand 2: CS computation for the complete closure considering reference points only in the fingertips, without considering the palm.

3.4. CS for precision grasps

Most of the soft hands described in the literature are designed to perform power grasps where fingers completely envelop the object. Using these hands to perform precision grasps is often an issue, due to the low number of controllable degrees of freedom (DoFs) and to the lack of accurate on-line proprioceptive measurements necessary to precisely place the fingertips over an object. In this article, we propose a CS-based approach to achieve precision grasps using soft hands. One possibility is to use only part of the hand and exploit the environment, as in the previously described lateral pinch (Figure 3b). Another possibility is to adjust hand preshape and distance from the object, in such a way that the hand touches it with the fingertips, achieving a precision grasp (Figure 6a).

Fig. 6.

Pisa/IIT SoftHand: partial closure for precision grasps. (a) Example of pinch grasp performed with this closing motion. (b) Initial configuration (partially closed hand) and final configuration (fully closed hand) of the partial closure motion.

To describe a precision grasp in terms of CS, it is important to choose a suitable closing motion. We can either track the motion of a part of the hand (e.g., index and thumb for pinch grasps), similarly to what we did for the lateral pinch with the RBO Hand 2, or track a partial closure of the hand, where the hand closes starting from a partially closed configuration (see, for example, Figure 6b). The first strategy considers only fingers that are important for precision grasps (e.g., thumb and index in the case of small objects), thus leading to having the center of the CS translated towards them. Hand pre-shaping is important to guarantee contacts at the fingertips. Approaching the object with the hand partially closed, in fact, allows a small closure to be performed near the object, without executing the whole closure, which might cause undesirable collisions with the object, often reorienting it and causing a grasp failure.

In this work, we combined the two approaches to compute the CS for a closing motion of the Pisa/IIT SoftHand that leads to a precision grasp.

We tracked the trajectories of the fingers in the Gazebo simulation of the hand. The closure motion that we considered starts with the hand partially closed (half of its range of motion), and ends with the hand fully closed (see Figure 6b). Two CSs related to this motion are depicted in Figure 7. Figure 7a shows the resulting CS when considering reference points in all the fingertips, whereas Figure 7b shows the resulting CS when considering reference points only in the thumb (fingertip, proximal phalanx) and the index (fingertip, middle phalanx). The latter was used in Experiment 3 to plan precision grasps with the Pisa/IIT SoftHand.

Fig. 7.

Pisa/IIT SoftHand: CS computation ( $o_{h}$ , $v_{CS}$ ) for performing precision grasps. The hand is a left hand and is seen from the top. The points are expressed in a reference frame placed on the hand wrist, with $x$ -axis pointing towards the middle finger, $y$ -axis pointing in the opposite direction with respect to the thumb, and $z$ -axis exiting from the plane. Magenta circles and red stars indicate initial and final positions of the reference points, respectively. CS for the motion illustrated in Figure 6b, (a) when considering reference points placed in all the fingertips and (b) when considering reference points placed in the fingertip and proximal phalanx of the thumb and fingertip and middle phalanx of the index.

Assuming a reference frame placed on the wrist with the $x$ -axis pointing towards the middle finger, $y$ -axis pointing in the opposite direction with respect to the thumb, and $z$ -axis exiting from the plane, in the first case $v_{CS} = [0.87, 0.5, - 0.03]$ mm and $o_{h} = [104, 2, - 16]$ mm, in the second $v_{CS} = [0.74, 0.67, - 0.01]$ mm and $o_{h} = [98, - 36, - 31]$ mm. While the orientation of the CS does not vary significantly in the two cases, if only a subset of fingers is considered, the center of the CS translates and becomes particularly useful for performing precision grasps.

4. Grasp planning

Our grasp planner is composed of two modules, one object-centric and the other hand-centric. The former module is related to grasp planning with the aid of AI, and we named it grasp quality convolutional neural network (GQ-CNN) module. Given a depth image of the object to be grasped, it exploits all the encapsulated information, and provides an on-line estimate of the optimal grasp (center and direction) to be performed. The latter module is related to soft hand modeling, and we termed it CS module. Chosen a robotic hand, it computes off-line the functional model of a certain closure motion (i.e., the previously defined CS), thus providing an estimate of the best way to align the hand to an object for performing top grasps. As shown in Figure 8, and detailed in the following, the two modules are put together to align the hand to the object. Then, the grasp can be executed.

Fig. 8.

Block scheme of the proposed approach.

As we mentioned previously, the idea behind this module integration is that the CS allows a hand closure to be approximated with a direction, and this direction can be thought of as the closing direction of an equivalent parallel-jaw gripper. In other words, the CS abstraction allows to use a planner that was trained for parallel-jaw grippers without retraining it, even if the end-effector is different.

In the proposed approach, the softness of the hand is not modeled explicitly, but it is the condition that actually allows us to apply such a simplified grasp planning strategy. The passive adaptability typical of soft hands allows us to compensate for the adopted simplifications (grasp planner for rigid parallel-jaw gripper, hand modeling as a jaw gripper), to cope with uncertainties, and to safely interact with the environment. This would not be possible with rigid hands, for which even small uncertainties in the fingers positioning might lead to unwanted and dangerous collisions with the environment (Brock et al., 2016).

4.1. GQ-CNN module

The GQ-CNN has been presented by Mahler et al. (2017). It is the core component of the so-called CEM-augmented DexNet 2.0 grasp planner, which can be used for grasp planning, since it provides an estimation of the most robust grasp under uncertainties in sensing and control. The planner considers a parallel-jaw gripper performing top grasps of objects resting on a planar surface. Given a depth image of the object to be grasped, a set of grasp candidates is sampled with the rejection sampling algorithm presented in Mahler et al. (2016), and the robustness of each candidate is evaluated by the neural network. This step is shown in the first image in Figure 9, where the different colors correspond to different robustness values. After this preliminary step, the most robust grasps, i.e., the “elite grasps”, are selected. Then, the cross-entropy method (CEM) iteratively computes a Gaussian mixture model of the most robust grasps, and new grasp candidates are sampled from this distribution (“sampled grasps”). This procedure is iterated for a predefined number of times, after which the most robust grasp is selected (see Figure 9). Its center ( $o_{NN}$ ) and direction ( $v_{NN}$ ) are returned as outputs of the planner. The image of the object to be grasped is acquired thanks to an overhead depth camera, and the estimation of the grasp candidate robustness is achieved by feeding the GQ-CNN with the candidate center depth $z$ relative to the camera and the depth image. The grasp robustness is computed by means of the so-called robustness function $Q$ , which, given the grasp $u$ and the point cloud $y$ of the object, returns the probability of the grasp success $S$ :

Q (u, y) = E [S | u, y]

(5)

Fig. 9.

Subsequent steps for finding the most robust grasp.

Uncertainties in sensing are included in the problem formulation, as explained in Mahler et al. (2017).

The approximation $Q_{θ} (u, y)$ of the function $Q$ was learned by the GQ-CNN, and the evaluation on the grasp candidates allows to apply a grasping policy selecting the grasp that maximizes the success:

π_{θ} (y) = \underset{u \in C}{\arg \max} Q_{θ} (u, y)

where $C$ is a discrete set of grasps respecting some constraints. Learning the robustness function $Q$ instead of the policy $π$ allows enforcing specific task constraints without the necessity of retraining the neural network. The GQ-CNN has been trained only on the synthetic data collected in the so-called Dexterity Network (Dex-Net) 2.0, which is a dataset consisting in $6.7$ million data points and associating parallel-jaw grasps with analytic grasp quality measures.

Although the neural network was trained only on synthetic data, the CEM-augmented version of the planner achieved a $94 %$ success rate (Mahler et al., 2017). A limitation of the approach proposed by Mahler et al. is that the grasp planner is thought only for a parallel-jaw gripper performing top grasps.

In addition to the grasp center $o_{NN}$ and direction $v_{NN}$ , this object-centered module outputs also the estimated width of the object in the grasp direction. We devised an algorithm that provides such estimate based on the inspection of the data captured by the depth image. First, a mask is applied to the raw depth data in order to store only the data related to the object to be grasped. Let us denote with $V$ the set of these valid points. Each element of $V$ contains the spatial coordinates of the pixel it refers to with respect to a reference system associated to the depth camera. Then, in the set of candidates $C$ , we store only the elements of $V$ satisfying the condition $| z - z_{G} | < ϵ$ and belonging to the line passing through $o_{NN} = [x_{G}, y_{G}, z_{G}]$ with angular coefficient related to the estimated grasping direction $v_{NN}$ .

Among the points stored in $C$ , i.e., all the points lying on the line, we consider only those that are antipodal with respect to the grasp center, and we search for the two that are at the maximum distance. This distance is considered to be the estimated size of the object.

4.2. CS module

The CS module that we have implemented for the experimental validation described in this article involves the off-line computation of the CS corresponding to:

the complete closure motion (heavy wrap) of the Pisa/IIT SoftHand (computed by Pozzi et al., 2018);

the complete closure motion (heavy wrap) of the RBO Hand 2 (computed in the previous section);

a partial closure motion of the RBO Hand 2 leading to a lateral pinch (computed in the previous section);

a partial closure motion of the Pisa/IIT SoftHand leading to a precision grasp (computed in the previous section).

It is worth remarking that, in general, if the considered robotic hand can perform more than one closure motion (e.g., power and precision grasps), a list of CSs can be associated to the hand. In this case, the CS module could automatically perform an on-line selection of the most suitable CS to be used for grasp execution. The choice could be based on object size, computed, for example, as explained in the previous subsection. To select the best CS, one could either leverage intuitive rules as those that we applied in the experiments conducted for this article (e.g., heavy wrap for objects with a size that is comparable with that of the hand, lateral pinch and precision grasp for smaller objects), or rely on an algorithm that compares the object size with a certain size associated with the CS. A strategy could be as follows: among the CSs with sizes that are bigger than the object size, the CS whose size is the closest (or second closest) to the object size would be a good candidate for the grasp. Research on establishing this “CS size”, however, is still on-going, since it is not trivial to identify a general way of evaluating the size of a closure only based on the motion of the reference points, and taking into account that the hand has to achieve a certain orientation over the object.

4.3. Modules integration: grasp planning with soft hands

The CS provides a simplified way to treat soft hands characterizing a specific closing motion that the hand can achieve through a direction of maximum closure applied at a certain point. The obtained center and direction can be thought as the grasping center and direction of a parallel-jaw gripper. This is the fundamental principle of the proposed integration between the GQ-CNN module and the CS module. We use the term planning with soft hands for the resulting module, which was then integrated with the grasp execution module, allowing object approaching and grasping (Figure 8).

The GQ-CNN output data and the CS hand model are combined to perform the so-called CS-alignment, i.e., the alignment between the hand CS and the grasp angle and center retrieved by the neural network. In general, this can be achieved by computing the transformation $T_{CS}$ that brings $o_{h}$ on $o_{NN}$ and aligns $v_{CS}$ with $v_{NN}$ (see Figure 10).

Fig. 10.

General CS-alignment procedure.

Let us consider a world reference frame ${W}$ that has the $xy$ -plane parallel to the surface over which the object is lying, and the $z$ -axis exiting from it. Since the GQ-CNN outputs assume that the direction of closure of the gripper is always parallel to the table, the fist step is to align the CS parallel to the $xy$ -plane.

Let us indicate with $α$ the angle between CS and the $xy$ -plane, and with $θ$ the angle between CS projection on $xy$ -plane and $v_{NN}$ , these angles are not shown in Figure 10 for the sake of clarity. The overall transformation $T_{CS}$ to be imposed to the hand reference frame to perform the CS-alignment will result from the combination of a rotation of $α$ with respect to an axis parallel to the plane $xy$ , a rotation of $θ$ with respect to the $z$ -axis, and a translation between $o_{h}$ and the center of the grasp $o_{NN}$ .

5. Experimental results

The experimental validation is divided into three experiments. Experiment 1 evaluates the capabilities of the grasping policy integrating CS-alignment and GQ-CNN module in terms of success rate over a large object dataset. This is demonstrated using the Pisa/IIT SoftHand for performing heavy wrap power grasps. Experiment 2 assesses the applicability of the proposed approach to heavy wraps performed with a different soft robotic hand, the RBO Hand 2. Results obtained with our grasp planner are compared with a baseline method, called S-alignment. Experiment 3 tests the functioning of the devised grasp planner for performing grasps that are not heavy wraps: lateral pinch and precision grasp.

While the advantages of using the CS when planning heavy wraps with the Pisa/IIT SoftHand, compared with aligning the hand straight with the object, has already been established in Pozzi et al. (2018) for a variety of objects, Experiment 1 aims to verify the success of its integration with the GQ-CNN module. The purpose of Experiment 3 is to show that the CS-alignment also works for grasps that are less commonly implemented with soft hands. As, to the best of the authors’ knowledge, there is no well-established baseline in the literature for such types of grasps performed with soft hands, comparison with other strategies remains an open problem.

5.1. Experimental setup

We performed experiments either with a 7-DoF Sawyer collaborative robot arm (Rethink Robotics) or with a LBR iiwa (KUKA AG), having an ATI Gamma $6$ -axis force–torque sensor (ATI Industrial Automation, Inc.) and either a Pisa/IIT SoftHand (Catalano et al., 2014) or a RBO Hand 2 (Deimel and Brock, 2016) mounted on its end-effector (Figure 11). The depth image of the object to be grasped was acquired with an Asus Xtion PRO Live camera (ASUSTeK Computer Inc.) positioned in front of the robot, at a distance from the tabletop in the range 55–70 cm, similarly to Mahler et al. (2017). At the beginning of each experimental session, the camera was calibrated with the aid of a chessboard, which then was used just for a visual check of the neural network output. We used ROS to implement robot controllers and information exchange between devices (Quigley et al., 2009). Experiments ran on a notebook having the Intel Core i7-8750H as processor and the Nvidia GeForce GTX 1050 as GPU. On average, the GQ-CNN module returned an output in 0.9 s.

Fig. 11.

Experimental setup: (a) with Pisa/IIT SoftHand; (b) with RBO Hand 2. The RGB-D camera detects the object on the table and the robotic arm equipped with a force/torque sensor and a soft hand executes the grasp.

The two soft hands that we used have different features. The Pisa/IIT SoftHand is an underactuated tendon-driven anthropomorphic robotic hand with one motor and 19 DoFs, designed to reproduce the first human hand synergy (Santello et al., 1998). The RBO Hand 2 is an intrinsically soft hand made of seven pneumatic actuators, five for the fingers and two for the palm (Deimel and Brock, 2016). The actuators are made of silicon rubber, are reinforced by a 3D printed structure, and are surrounded by passive wires that constrain their motion. The model that we used has four degrees of actuation, i.e., four channels that can be inflated separately. One channel is for the little, ring, and middle fingers, one for the index, one for the thumb and the inner part of the palm and one for the outer part of the palm.

We tested our grasping policies on objects having different weights and shapes. Adversarial surface properties, e.g., reflectivity, smoothness, softness, and roundness, were also considered. Depending on the performed experiment, we used all or a subset of the objects that are shown in Figure 12, and whose properties are listed in Table 1.

Fig. 12.

Objects set (from back to front, from left to right): hollow box, sugar box, chips can, funnel, teddy bear, cuboid box, big funnel, plastic bowl, pudding box, bowling pin, strawberry, Lego Duplo piece, small cube, and plastic toy.

Table 1.

Physical properties of the objects used in the experiments. The chips can, the pudding and sugar boxes, the small cube, the Lego Duplo piece, and the strawberry are taken from the YCB Dataset (Calli et al., 2015). Note that $d$ and $h$ indicate the diameter and the height of the object, respectively.

Object	Size (mm)	Weight (g)
Chips can	$d = 75$ , $h = 250$	$205$
Cuboid box	$200 \times 100 \times 40$	$51$
Sugar box	$175 \times 89 \times 38$	$514$
Pudding box	$110 \times 89 \times 35$	$187$
Funnel	$d = 140$ , $h = 140$	$45$
Bowling pin	$h = 170$	$17$
Big funnel	$d = 150$ , $h = 70$	$96$
Plastic bowl	$d = 200$ , $h = 75$	$84$
Hollow box	$220 \times 165 \times 45$	$52$
Plastic toy	$h = 90$	$66$
Teddy bear	$h = 170$	$111$
Small cube	$h = 25$	$8$
Lego Duplo piece	$42 \times 30 \times 30$	$13$
Strawberry	$55 \times 43.8$	$18$
Small vial	$d_{\max} = 20, h = 50$	$15$

The cylindrical chips can and the cuboid boxes are objects with prototypical shapes. The funnel and the bowling pin are interesting because they can easily roll and generate unwanted dynamic behaviors when touched by the hand, owing to their roundness, non-regular shape, and lightness. The plastic bowl, the hollow box, and the big funnel are objects that should be picked up from their edges. The approach followed in Pozzi et al. (2018), based on building the bounding box of the entire object, would lead to failures in this case, because the grasp center would end up in the hollow part of the object. The plastic toy was chosen as it has a rather irregular surface. The teddy bear was selected because it is made of compliant material, but it is not soft and lightweight enough to undergo large deformations modifying its shape and spatial configuration when the hand touches it. The occurrence of such deformations would lead to results possibly unrelated to the output of the planning.

The first 11 objects were used in Experiment 1, and a subset of them was used in Experiment 2, as they are suitable to be grasped with heavy wraps. The small cube, the Lego Duplo piece, the strawberry, and the small vial are relatively small objects with respect to the used hands and might require delicate handling (strawberry). This is why they were used in Experiment 3, when testing lateral pinch and precision grasps.

Note that the chips can, the pudding and sugar boxes, the small cube, the Lego Duplo piece, and the strawberry are standard objects taken from the YCB Dataset (Calli et al., 2015).

5.2. Experiment 1: Evaluation of the modules integration

The aim of Experiment 1 was to evaluate the CS exploitation integrated with a grasp planner specifically designed for parallel-jaw grippers. We performed top-grasp experiments with a right Pisa/IIT SoftHand attached to a Sawyer robot arm to show the feasibility of the integrated approach shown in Figure 8, and to validate its robustness over multiple trials.

The hand–object alignment was executed at fixed height from the tabletop using the motion planning provided by MoveIt.² The robot motion was then controlled by means of a feedback-driven state machine, and the hand closed when a certain force threshold was exceeded due to the occurred contact between the hand and the object. During experiments, objects were placed in a random pose in the robot workspace.

Figure 13 shows examples of planned grasps and Table 2 reports the results obtained for the chosen objects in terms of success rate.³ A grasp was considered successful when the hand could stably hold the object while the arm was lifting for $6$ seconds with a velocity of $3$ cm/s. For each object, $10$ grasp attempts were performed. The overall success rate was $97 / 110$ ( $~ 88 %$ ).

Fig. 13.

Examples of planned grasps for some of the chosen objects.

Table 2.

Pisa/IIT SoftHand: Heavy wrap top grasps performed with the proposed integrated strategy. Each object was grasped 10 times. The number of successes is reported in the table. The total success rate was $~ 88 %$ .

Object	# Successes
Chips can	10/10
Cuboid box	10/10
Sugar box	9/10
Pudding box	10/10
Funnel	8/10
Bowling pin	6/10
Big funnel	8/10
Plastic bowl	10/10
Hollow box	7/10
Plastic toy	10/10
Teddy bear	9/10

5.3. Experiment 2: Application of the method to another soft hand

To test whether the proposed grasp planning strategy can be used also with other robotic hands, we applied it to the RBO Hand 2. First, we computed the CS of the complete closure of the RBO Hand, as explained previously. Then, we performed top grasps over four different objects, representative of three classes of objects that are present in Table 1: those with a prototypical shape (chips can, cuboid box), those that easily roll over the table top (bowling pin), and those that have a hollow part (bowl).

To investigate whether exploiting the RBO Hand 2 CS-alignment allows better grasping performance, we compared it with the baseline hand alignment that was used also in Pozzi et al. (2018). Such baseline has been called S-alignment (i.e., straight alignment) and consists of aligning the middle finger parallel to the chosen direction on the object, similarly to Bonilla et al. (2015) and Pozzi et al. (2018). The total success rate of the CS-alignment is $90 %$ , and of the S-alignment is $42.5 %$ . Results for the single objects are reported in Table 3.

Table 3.

RBO Hand 2: Heavy wrap top grasps performed with the proposed integrated strategy. Each object was grasped 10 times. The number of successes for the CS-alignment and the S-alignment is reported in the table. The total success rate of the first is $90 %$ and of the second is $42.5 %$ .

Object	CS-alignment	S-alignment
Chips can	10/10	0/10
Cuboid box	9/10	6/10
Bowling pin	7/10	3/10
Plastic bowl	10/10	8/10

During the experiments, objects were placed in a random pose in the robot workspace and the hand–object alignment was executed at fixed height from the tabletop using MoveIt. Then, the hand approached the object at a fixed velocity, until a certain force threshold was reached at the wrist. This indicated that a contact with the object occurred and the hand started closing over the object. A grasp was considered successful if the hand was able to hold the object while the arm was lifting for $6$ seconds with a velocity of $3$ cm/s.

5.4. Experiment 3: Other types of grasps

5.4.1. Lateral pinch

We implemented another type of power grasp with the RBO Hand, a lateral pinch (see Figure 3b), using the CS computed as in Figure 4b. We tested the strategy with two cuboid objects with different sizes (small cube and Lego Duplo piece) and one round object (strawberry). We observed that with the strawberry and the Lego Duplo piece, the grasp was always successful, while it always failed with the small cube (see Table 4). The trends were so clear that we limited the number of trials per object to five. The grasp execution was implemented as for the heavy wrap presented previously: the hand started closing when a certain force threshold was overcome. In this case, such event was determined by the contact with the table surface, that allowed the hand to slide and successfully pick up small objects.

Table 4.

RBO Hand 2, lateral pinch. Top grasps performed with the proposed integrated strategy. Each object was grasped five times.

Object	CS-alignment
Small cube	0/5
Lego Duplo piece	5/5
Strawberry	5/5

5.4.2. Precision grasp

To show that the CS model can also be exploited for closures that lead to precision grasps, we tested the previously computed CS for the index and the thumb of Pisa/IIT SoftHand to plan precision grasps over a small cube and a small vial (see Table 1 for their properties). In this case, we chose two very small objects, requiring a precision grasp, one with a squared base and one with a round base. The hand pre-shaping and the height from the table were selected according to the object size. In particular, the hand was programmed to start closing at a height such that the fingers almost touched the table. Results in terms of success rates are reported in Table 5. In this case, we used a left Pisa/IIT SoftHand attached to a KUKA LBR iiwa robot arm.

Table 5.

Pisa/IIT SoftHand, precision grasp. Top grasps performed with the proposed integrated strategy. Each object was grasped 10 times.

Object	CS-alignment
Small cube	7/10
Small vial	8/10

6. Discussion of the results

6.1. Experiment 1: The module integration works well for planning top grasps with soft hands

The grasping task that we consider is mainly composed of two complementary sub-tasks: grasp planning (i.e., the problem of identifying which are the best contact point locations satisfying a given criterion) and the grasp execution (which relies on the closing motion of the chosen hand). The two parts have already been validated separately. The GQ-CNN module providing the grasp planning has been compared by Mahler et al. (2017) with other methods, namely image-based grasp quality metric, point-cloud registration, random forest, and support vector machine. The CS module providing the functional hand model involved in the grasp execution has been tested by Pozzi et al. (2018), by comparing the CS-alignment with the so-called straight alignment.

In this work, we show that the integration between the two modules not only is feasible, but also allows grasping objects that would have been impossible to grasp with the simple bounding-box-based grasp planner used in Pozzi et al. (2018). Planning how to grasp an object based only on its bounding box may lead to an oversimplification of the object, and cannot be successfully applied to object categories such as hollow objects, because the hand would end up on the empty part of the object instead of on an edge.

Failures and successes observed during the experimental trials depend on the interaction between the hand and the object, on the setup features (hand characteristics, robot workspace, etc.), and on the planned grasp. Examples of successful and unsuccessful grasps are shown in Figures 14 and 15, respectively.

Fig. 14.

Some of the successful grasps with the Pisa/IIT SoftHand.

Fig. 15.

Some of the unsuccessful grasps with the Pisa/IIT SoftHand.

Our approach was always successful for the objects with a prototypical shape. This is something that was expected for two reasons: (i) for these objects, the neural network typically outputs a grasping point that is very close to the object center of symmetry as seen from the top; and (ii) results described in Pozzi et al. (2018) show that, with the CS-alignment, regular objects are picked up most of the time. There was only one failure on the sugar box, owing to its weight. In addition the plastic toy was grasped as if it was a small box, and its irregular surface did not prevent grasps from being successful.

A high success rate was obtained also for objects that can be picked up from edges, i.e., the bowl, the hollow box, and the big funnel. Failures with the hollow box were due to collisions between hand wrist and object (see Figure 15), whereas those with the big funnel were due to the fact that the network output a grasp center on the small part of the funnel.

When more complex objects were considered, our approach still resulted successful most of the times, but additional considerations should be made.

The funnel was correctly grasped when either the fingers or the thumb were placed inside the cave part of the object (see Figure 15). One of the failures was due to the planned grasp, that was significantly laterally displaced from the center, thus leading to an unwanted object motion when the contact with the hand occurred. The other failure was caused by an accidental collision between the hand and the object, leading to a displacement of the funnel.

Bowling pin failures occurred when the grasp center was in proximity of the object neck (see Figure 15). In these cases, in fact, the hand entered in contact with the object in a region that was higher than the goal point, and remained stuck there. The force threshold was thus exceeded too soon and the hand closed without being in contact with the right grasp center and without caging the object.

For the teddy bear, the failure occurred because the planned grasp center was near to the ear, and this led to a lateral head displacement during the grasp execution.

As previously described, the GQ-CNN returns also a prediction of the grasp success probability, which is estimated by means of the learned grasp robustness function. By analyzing the performed grasps and the relative probability values, we found that, in general, the performance of our grasping system depends more on the relative geometric configuration between the hand and object, rather than on the predicted probability. In the following, we report the probability values for one of the objects with best grasp performance, i.e., the chips can, and the object with the worst performance, i.e., the bowling pin. For the chips can, the probability was always in the range $[0.76, 0.86]$ and this is coherent with the observed high success rate. For the bowling pin, instead, grasp performance is rather uncorrelated from the predicted probability, and strongly depends on the location of the planned grasp center (see Table 6). The discrepancy between predicted probability and what actually happened is clearly related to the fact that the probability is computed for grasping with a parallel-jaw gripper, while we used a robotic hand which was modeled as a parallel-jaw gripper to simplify the grasp planning. However, the Pisa/IIT SoftHand has a much more complex geometric structure, and this additional complexity may sometimes lead to increased grasp success, and sometimes cause unexpected failures.

Table 6.

Top grasps of the bowling pin: results, grasp center location and predicted probability of success. Results are more related to the grasp location than to the probability.

Trial #	1	2	3	4	5
Result	S	S	S	F	S
Location	Body	Body	Body	Neck	Body
Success probability	0.61	0.64	0.67	0.81	0.72
Trial #	6	7	8	9	10
Result	F	S	F	F	S
Location	Neck	Head	Neck	Neck	Body
Success probability	0.37	0.45	0.75	0.37	0.73

6.2. Experiment 2: The CS computation and use generalizes well to other soft hands

The Pisa/IIT SoftHand and the RBO Hand 2 are different devices with different possible motion capabilities. This is reflected in the CS computation of the whole hand closure (heavy wrap (Cutkosky and Kao, 1989)). While the first hand generates a deformation that remains mainly in the $xy$ -plane, the second produces a more complex 3D motion, not only due to the thumb and palm actuation, but also related to the specific way in which each finger closes, i.e., by depicting a sort of self-rolling spiral. If compared with the RBO Hand 2, the Pisa/IIT SoftHand has a less spiral-like closing motion, owing to the different actuation mechanism, resulting in a different desired alignment between the hand and the object. For the Pisa/IIT SoftHand, the angle $θ$ around the axis normal to the table is more relevant (see Figure 16). The RBO Hand 2 requires also a rotation about an axis parallel to the plane of the table. It could happen that for some physical constraints it is not possible to make both rotations, for example, due to possible collisions of the robot arm with the table while aligning the hand. In these cases it could be better to privilege the alignment with the angle $θ$ than the angle $α$ , because $θ$ is on the plane of the grasp. In other terms, aligning the hand with a wrong $θ$ , may increase the possible grasp failures. Aligning the hand with a wrong $α$ , instead, could be better compensated by the intrinsic compliance of the hand.

Fig. 16.

Pisa/IIT SoftHand CS-alignment for heavy wrap.

Examples of successful and unsuccessful grasps with the RBO Hand 2 are shown in Figures 17 and 18, respectively. The CS-alignment for the heavy wrap of the RBO Hand 2 is effective with prototypical objects, apart from a failure with the box which was due to the fingers touching the box and moving it before starting to close. In addition, the bowl was always grasped successfully. The bowling pin remains a difficult object to grasp (cf., results of Experiment 1), mainly due to the fact that sometimes the planned grasp center lies on the neck of the pin, or to object displacements resulting from unwanted collisions with the hand.

Fig. 17.

Examples of successful grasps with the RBO Hand 2.

Fig. 18.

Examples of failures with the RBO Hand 2.

Concerning the baseline method, we observed that (i) the performed grasps were not robust enough to stably lift the object, and (ii) regarding the chips can, the failures were due to its weight and to the fact that the palm could not envelop it properly, since the palm lied on the can and the finger rolling motion prevented the thumb from exploiting opposability.

6.3. Experiment 3: The CS can be used to plan different grasps with respect to heavy wraps

Experiment 3 shows that it is feasible to use the CS to model lateral pinches and precision grasps.

For the lateral pinch with the RBO Hand 2 there were two main factors influencing the grasp. First, the interaction with the environment allowed the caging of the object and was fundamental in order to achieve a successful grasp, as shown in Figure 19. Second, the height of the object determined the grasp success, as too small objects were not caged in time between the thumb and the index during the closing motion (see Figure 20). This is due to the fact that the fingers close with a certain velocity and with some delays between each other, so by tuning these parameters it could also be possible to obtain higher success rates.

Fig. 19.

Sequence of frames taken during the execution of a lateral pinch over a Lego Duplo piece.

Fig. 20.

Sequence of frames taken during the execution of a lateral pinch over a small cube.

When grasping small objects using a precision grasp even small uncertainties on the hand and object positions may lead to failures. This is why, particularly when using highly underactuated hands, it is important to consider a reduced set of fingers (e.g., thumb and index), and start from a partially closed initial position (CS depicted in Figure 7b). Preliminary experiments where we considered reference points in all the fingers (CS depicted in Figure 7a) led to unsuccessful grasps, as the center of the CS was too close to the middle of the palm and the objects were too small to be caged by the fingers. On the other hand, when considering just two fingers but starting from a completely open position, unwanted collisions with the objects occurred.

Figure 21 shows a successful grasp of a small cube. The hand starts from half of its total closure and then gently grasps the object between the fingertips of the thumb and index. Failures, such as that shown in Figure 22, were mainly due to misalignment between hand and object. In fact, the alignment of the planar components of the center of the CS to the center of the object is prone to uncertainties coming from the vision system and from the camera-to-robot calibration. When performing power grasps these are mitigated thanks to hand compliance and to the fact that the hand starts closing after the palm touches the object. In addition, in precision grasps, also uncertainties on the positioning in the direction normal to the plane are relevant.

Fig. 21.

Sequence of frames taken during the execution of a precision grasp over a small cube.

Fig. 22.

Sequence of frames taken during the execution of a precision grasp over a small vial.

6.4. General considerations

In addition to the discussion of experimental results, we conduct here a higher-level analysis of the proposed grasp planning approach. In particular, we want to characterize our system by means of the four key aspects of systems building suggested by Eppner et al. (2018), i.e., modularity versus integration, computation versus embodiment, planning versus feedback, and generality versus assumptions.

6.4.1. Modularity versus integration

Modularity allows a problem to be broken down in easier-to-solve sub-problems. As mentioned previously, our system is a result of a tight integration of different modules. The GQ-CNN module provides an estimation of the center and orientation of the most robust grasp, and the CS module provides a simple model of a closing motion of a chosen hand. The resulting integration is the grasp planning with soft hands module. Subsequently, it has been integrated with the grasp execution module, which, in turn, is given by the integration of a trajectory planning achieved with the standard module MoveIt and a force-feedback-guided state machine. Our system works on ROS, which allows us to achieve a great level of module integration. Note that the CS module plays a key role in the proposed method, as it allows the planning on the object to be decoupled, which can be performed with any grasp planner designed for parallel-jaw grippers, from the used end effector, whose closing motion can be described in terms of CS.

6.4.2. Computation versus embodiment

This aspect describes the relationship between software computation and the embodiment of some processes in the robot hardware in the system at hand. To characterize our system, we think that it is appropriate to split the entire process into two subsequent stages. The first stage includes the grasp planning and the hand alignment to the desired grasp, while the second stage includes only the hand closure. The first stage relies only on computation, while the second relies on the embodiment of the grasping capabilities of the soft hand, whose compliance allows safe interaction with the environment and high adaptability to objects.

6.4.3. Planning versus feedback

In Eppner et al. (2018), the difference between these aspects is centred on the type of solutions that they obtain: planning allows general solutions to be found, while feedback is involved in local solution search. Our approach is closer to planning than to feedback. In fact, after the acquisition of the object depth image, grasp planning, and robot end-effector alignment work without explicitly needing feedback from the scene. In Experiments 1 and 2, during the object approach, the execution is feedback-driven, since the robot motion stops and the grasp begins when the force exerted on the object exceeds a certain threshold.

6.4.4. Generality versus assumptions

When building robotic systems, it is crucial to find a trade-off between the generality of the proposed solution and the presence of underlying assumptions (Eppner et al., 2018). The assumption that mainly affects our system is that the neural network is trained for executing grasps with a parallel-jaw gripper. This assumption is clearly too restrictive in the case of grasping using a multi-fingered soft hand. However, modeling the hand through the CS, overcomes this difficulty, showing high generalizability to different robotic hands.

7. Conclusions

In this paper, we propose a general method for grasping objects from the top with soft robotic hands. The strength of the approach resides in the closure signature (CS), i.e., a model that associates a preferred grasping direction to a closure motion performed by a robotic hand. The CS is general enough to model power and precision grasps and its computation requires only the tracking of some chosen reference points on the hand during the motion that one wants to represent. The information provided by the CS can be used to suitably align the hand with the output of a grasp planner prescribing the best way to grasp a certain object (CS-alignment).

To validate the feasibility of the proposed approach, we tested different CSs of two different robot hands in combination with an AI-based grasp planner. Multiple experimental trials showed that: (i) the proposed integration generates successful power grasps from the top over a variety of objects (Experiment 1); (ii) the CS-alignment generalizes well to heavy wraps performed with different soft hands (Experiment 2); (iii) the CS model can be used also to perform grasps that are not heavy wraps, such as lateral pinch and precision grasp (Experiment 3). The road towards a dexterous use of soft hands is still long. Without a specific planner for soft hands, the complex interaction between hand, environment, and object largely determines the final obtained grasps. In this work, we have shown that the use of the CS model, that is computed for a certain intended closing motion, possibly combined with hand pre-shaping, allows us to plan the emerging grasp type.

The field of soft manipulation poses several challenges, regarding both the design of new hands and their control. There is a growing need of mathematical tools and shared standards to tackle these problems, devising methods general enough to be applicable to different robotic setups. However, it is important to find a balance between this quest for generality and the focus on specific device characteristics, because, in soft hands, design, control, and planning are inevitably strictly related. The CS follows this philosophy, as it can be computed for different robotic hands and, by modeling closure motions, it is able to describe the functioning of the chosen device. In the planning phase, the CS-alignment gives a way to increase grasp success independently from the adopted gripper, but the hand intrinsic compliance is exploited to gain adaptability to the object surface, safe interaction with the environment, and robustness to uncertainties coming from sensor inaccuracies.

Future work will focus on further applications of the presented results in real-world scenarios, as well as on their extension to deal with different problems, including environmental constraints exploitation. In a recent work, we already showed that the CS represents an important tool for well aligning the hand to the object in grasping strategies involving environmental constraints exploitation (Bimbo et al., 2019). We will also investigate the possibility of explicitly considering the interaction with the environment within the CS framework. In addition, the CS could play an important role in the open quest for standards and benchmarks in robotic manipulation.

Footnotes

Acknowledgements

We gratefully acknowledge Professor Oliver Brock and the members of the Robotics and Biology Laboratory of the Technische Universität of Berlin for having provided us with the data describing the trajectories of the reference points placed on the RBO Hand 2.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We gratefully acknowledge the funding provided by the SOMA project (European Commission, Horizon 2020 Framework Programme, H2020-ICT-645599).

Notes

ORCID iDs

Maria Pozzi

Joao Bimbo

References

Barequet

Har-Peled

(2001) Efficiently approximating the minimum-volume bounding box of a point set in three dimensions. Journal of Algorithms 38(1): 91–109.

Bicchi

(1995) On the closure properties of robotic grasping. The International Journal of Robotics Research 14(4): 319–334.

Bimbo

Turco

Ghazaei Ardakani

, et al. (2019) Exploiting robot hand compliance and environmental constraints for edge grasps. Frontiers in Robotics and AI 6: 135.

Birglen

Lalibertè

Gosselin

(2008) Underactuated Robotic Hands (Springer Tracts in Advanced Robotics, Vol. 40). Berlin: Springer.

Bohg

Morales

Asfour

Kragic

(2014) Data-driven grasp synthesis – a survey. IEEE Transactions on Robotics 30(2): 289–309.

Bonilla

Resasco

Gabiccini

Bicchi

(2015) Grasp planning with soft hands using bounding box object decomposition. In: Proceedings IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 518–523.

Borst

Fischer

Hirzinger

(1999) A fast and robust grasp planner for arbitrary 3D objects. In: Proceedings IEEE International Conference on Robotics and Automation (ICRA), Vol. 3. IEEE, pp. 1890–1896.

Brock

Park

Toussaint

(2016) Mobility and manipulation. In: Siciliano

Khatib

(eds.) Springer Handbook of Robotics. Springer Science & Business Media, pp. 1007–1030.

Calli

Walsman

Singh

Srinivasa

Abbeel

Dollar

(2015) Benchmarking in manipulation research: Using the Yale–CMU–Berkeley object and model set. IEEE Robotics Automation Magazine 22(3): 36–52.

10.

Catalano

Grioli

Farnioli

Serio

Piazza

Bicchi

(2014) Adaptive synergies for the design and control of the Pisa/IIT SoftHand. The International Journal of Robotics Research 33(5): 768–782.

11.

Choi

Schwarting

DelPreto

Rus

(2018) Learning object grasping for soft robot hands. IEEE Robotics and Automation Letters 3(3): 2370–2377.

12.

Cutkosky

(1989) On grasping choice, grasp models, and the design of hands for manufacturing tasks. IEEE Transactions on Robotics and Automation 5(3): 269–279.

13.

Cutkosky

Kao

(1989) Computing and controlling compliance of a robotic hand. IEEE Transactions on Robotics and Automation 5(2): 151–165.

14.

Deimel

Brock

(2016) A novel type of compliant and underactuated robotic hand for dexterous grasping. The International Journal of Robotics Research 35(1–3): 161–185.

15.

Dollar

Howe

(2010) The highly adaptive SDM hand: Design and performance evaluation. The International Journal of Robotics Research 29(5): 585–597.

16.

Eppner

Deimel

Álvarez Ruiz

Maertens

Brock

(2015) Exploitation of environmental constraints in human and robotic grasping. The International Journal of Robotics Research 34(7): 1021–1038.

17.

Eppner

Höfer

Jonschkowski

, et al. (2018) Four aspects of building robotic systems: Lessons from the Amazon Picking Challenge 2015. Autonomous Robots 42(7): 1459–1475.

18.

Gioioso

Salvietti

Malvezzi

Prattichizzo

(2013) Mapping synergies from human to robotic hands with dissimilar kinematics: an approach in the object domain. IEEE Transactions on Robotics 29(4): 825–837.

19.

Goldman

(1991) Recovering the Data from the Transformation Matrix, GEMS, volume VII-2. New York: Academic Press.

20.

Gupta

Eppner

Levine

Abbeel

(2016) Learning dexterous manipulation for a soft robotic hand from human demonstrations. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 3786–3793.

21.

Haas-Heger

Iyengar

Ciocarlie

(2018) Passive reaction analysis for grasp stability. IEEE Transactions on Automation Science and Engineering 15(3): 955–966.

22.

Hang

Morgan

Dollar

(2019) Pre-grasp sliding manipulation of thin objects using soft, compliant, or underactuated hands. IEEE Robotics and Automation Letters 4(2): 662–669.

23.

Higham

Schreiber

(1990) Fast polar decomposition of an arbitrary matrix. SIAM Journal on Scientific and Statistical Computing 11(4): 648–655.

24.

Hirose

Umetani

(1978) The development of soft gripper for the versatile robot hand. Mechanism and Machine Theory 13(3): 351–359.

25.

Huebner

Ruthotto

Kragic

(2008) Minimum volume bounding box decomposition for shape approximation in robot grasping. In: 2008 IEEE International Conference on Robotics and Automation. IEEE, pp. 1628–1633.

26.

Hughes

Culha

Giardina

Guenther

Rosendo

Iida

(2016) Soft manipulators and grippers: A review. Frontiers in Robotics and AI 3: 69.

27.

Lenz

Lee

Saxena

(2015) Deep learning for detecting robotic grasps. The International Journal of Robotics Research 34(4–5): 705–724.

28.

Levine

Pastor

Krizhevsky

Ibarz

Quillen

(2018) Learning hand–eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research 37(4–5): 421–436.

29.

Mahler

Liang

Niyaz

, et al. (2017) Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In: Robotics: Science and Systems (RSS).

30.

Mahler

Matl

Liu

Gealy

Goldberg

(2018) Dex-Net 3.0: Computing robust robot suction grasp targets in point clouds using a new analytic model and deep learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5620–5627.

31.

Mahler

Matl

Satish

, et al. (2019) Learning ambidextrous robot grasping policies. Science Robotics 4: 26.

32.

Mahler

Pokorny

Hou

, et al. (2016) Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a multi-armed bandit model with correlated rewards. In: IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 1957–1964.

33.

Miller

Knoop

Christensen

Allen

(2003) Automatic grasp planning using shape primitives. In: Proceedings IEEE International Conference on Robotics and Automation (ICRA), Vol. 2, pp. 1824–1829.

34.

Myronenko

Song

(2010) Point set registration: Coherent point drift. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(12): 2262–2275.

35.

Pinto

Gupta

(2016) Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3406–3413.

36.

Pozzi

Malvezzi

Prattichizzo

(2017) On grasp quality measures: Grasp robustness and contact force distribution in underactuated and compliant robotic hands. IEEE Robotics and Automation Letters 2(1): 329–336.

37.

Pozzi

Salvietti

Bimbo

Malvezzi

Prattichizzo

(2018) The closure signature: A functional approach to model underactuated compliant robotic hands. IEEE Robotics and Automation Letters 3(3): 2206–2213.

38.

Prattichizzo

Malvezzi

Gabiccini

Bicchi

(2013a) On motion and force controllability of precision grasps with hands actuated by soft synergies. IEEE Transactions on Robotics 29(6): 1440–1456.

39.

Prattichizzo

Malvezzi

Gabiccini

Bicchi

(2013b) On motion and force controllability of precision grasps with hands actuated by soft synergies. IEEE Transactions on Robotics 29(6): 1440–1456.

40.

Quigley

Conley

Gerkey

, et al. (2009) ROS: An open-source robot operating system. In: ICRA Workshop on Open Source Software, p. 5.

41.

Roa

Suarez

(2015) Grasp quality measures: Review and performance. Autonomous Robots 38(1): 65–88.

42.

Salvietti

(2018) Replicating human hand synergies onto robotic hands: A review on software and hardware strategies. Frontiers in Neurorobotics 12: 27–31.

43.

Salvietti

Malvezzi

Gioioso

Prattichizzo

(2014) On the use of homogeneous transformations to map human hand movements onto robotic hands. In: IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, pp. 5352–5357.

44.

Salvietti

Malvezzi

Gioioso

Prattichizzo

(2015) Modeling compliant grasps exploiting environmental constraints. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 4941–4946.

45.

Santello

Flanders

Soechting

(1998) Postural hand synergies for tool use. The Journal of Neuroscience 18(23): 10105–10115.

46.

Saxena

Driemeyer

(2008) Robotic grasping of novel objects using vision. The International Journal of Robotics Research 27(2): 157–173.

47.

Siciliano

Sciavicco

Villani

Oriolo

(2009) Robotics: Modeling, Planning and Control. Berlin: Springer.

48.

Wang

Ling

(2016) Robotic grasp detection using deep learning and geometry model of soft hand. In: 2016 IEEE International Conference on Consumer Electronics - China (ICCE-China), pp. 1–6.