Exploitation of environmental constraints in human and robotic grasping

Abstract

We investigate the premise that robust grasping performance is enabled by exploiting constraints present in the environment. These constraints, leveraged through motion in contact, counteract uncertainty in state variables relevant to grasp success. Given this premise, grasping becomes a process of successive exploitation of environmental constraints, until a successful grasp has been established. We present support for this view found through the analysis of human grasp behavior and by showing robust robotic grasping based on constraint-exploiting grasp strategies. Furthermore, we show that it is possible to design robotic hands with inherent capabilities for the exploitation of environmental constraints.

Keywords

Grasping robotic hands human grasping soft hands environmental constraints constraint exploiting grasp strategy

1. Introduction

Humans are excellent graspers. Despite decades of research on robotic grasping, we have yet to establish the same level of competency in robotic systems. What lets humans grasp so well? There are many answers to this question: most are associated with active research areas in robotics. We propose that human grasp performance is to a significant extent the result of carefully orchestrated interactions between the hand, the object, and the environment. Our premise is the following: a competent grasper must exploit constraints present in the environment by employing physical contact so as to counteract uncertainty in state variables most relevant to grasp success. If this premise is true, robust and versatile grasping is the process of determining sequences of motions that take advantage of these constraints in the most effective manner.

Although the observation of human grasping intuitively supports our premise, because humans routinely establish contact with the environment when grasping, we are not aware of systematic studies on the use and purpose of such contacts in the psychology literature. We present a study on human grasping that evaluates the plausibility of our premise. Specifically, we investigate whether humans increase the amount of interaction with environmental constraints when uncertainty about the environment is increased through an induced visual impairment. For this, we establish a set of parameters to quantify the amount of interaction with the support surface during grasping, and test for effects on those parameters induced by the visual impairment.

Ongoing research on robotic grasping provides further support for our premise. Novel gripper and hand designs often include compliant materials or actuators. In our view, this does not only lead to more robust interactions between hand and the grasped object, but also facilitates the exploitation of environmental constraints. There are several studies of novel hands, reviewed in the next section, that deliberately exploit environmental constraints in specific application scenarios or for specific grasps. Research in grasp planning has also begun to consider the use of environmental constraints, however, either to a limited extent or in specifically tailored approaches. Beyond these instances, to the best of our knowledge, there is no comprehensive approach for the generic, orchestrated use of environmental constraints in robotic grasping.

In this paper, we outline the beginnings of an integrated research agenda towards robotic grasping by leveraging environmental constraints. This agenda spans the study of human grasping, the development of appropriate grasp strategies, the required perceptual strategies to determine when each of the strategies is most appropriate, and the design of robotic hands tailored for the exploitation of environmental constraints.

This journal paper is an extended version of the paper presented at the International Symposium on Robotics Research (Deimel et al., 2013). Changes include a more detailed analysis of the human grasping experiments and a new robotic grasping strategy that exploits another commonly occurring environmental constraint: the wall-constrained grasp.

2. Related work

To support our claim that competent graspers exploit environmental constraints, we divide related work into three categories based on the types of interactions they consider. The first category, which also marks the beginnings of grasping research in robotics, analyzes quasistatic grasps and thus does not exploit any interaction that might occur during the grasping process. The second category exploits interactions between hand and object. The final and most recent category exploits interactions between hand, object, and environment, enabling the consideration of environmental constraints for robust grasping.

2.1. Force closure

Early grasping research emphasizes the concepts of force and form closure, reflecting a static grasping relationship between hand and object (Mason, 2001; Prattichizzo and Trinkle, 2008). A grasp is commonly expressed as a set of disembodied point contacts. Physical interactions occurring during the grasp, and sometimes even the limitations that result from the kinematics of the hand, are often not accounted for during grasp planning. These approaches require detailed models of both the environment and the hand to exactly attain the planned grasping configuration.

This approach to grasping promotes the design of hardware by which precise placement of specific contact points on objects can be achieved. Consequently, the dominant paradigm of hand design leads to mechanically complex, rigid hands with many degrees of freedom (Kawasaki et al., 1999), some with compliant actuation (Grebenstein et al., 2012).

This line of research continues to be active and successful, as evidenced by a large number of sophisticated and capable grasp planners (Ciocarlie and Allen, 2009), simulators (Miller and Allen, 2004), and hand designs (Controzzi et al., 2014). In our experience, however, the grasps determined by these approaches do not reliably transfer to the real world when executed even on the most sophisticated hands. The fact that most classical grasp metrics only poorly reflect physical reality was also shown by Balasubramanian et al. (2012).

Interestingly, early studies of human grasping also followed this static view of grasping, largely ignoring the grasping process itself. This is reflected in grasp taxonomies, classifying grasps according to the final hand posture attained after the grasp process is completed (Cutkosky, 1989; Feix et al., 2009). Even the early work on postural synergies, which has had a profound impact on robotics, initially only considered synergies of static grasp postures (Santello et al., 1998). These studies do not capture the dynamic processes and the exploitation of environmental constraints we believe to be crucial for robust grasping.

2.2. Interactions between hand and object

During grasp execution, mechanical compliance in the hand leads to an adaptation of the hand’s configuration to the object’s shape. This shape adaptation aids grasping performance by compensating uncertainty in sensing, actuation, and the world model. This benefit is realized to a large extent through the attainment of many contact points, most of which would not have been found by a static grasp planner. Shape adaptation therefore significantly increases the chances of achieving force closure with a grasp. Much of the recent work in robotic grasping attempts to leverage this effect explicitly, especially in hand design. The positive pressure gripper (Amend et al., 2012) represents an extreme example in this regard. It uses granular material enclosed in a flexible bag to achieve compliance of the entire gripper to large parts of the object’s geometry. By evacuating the air contained in the bag and thereby jamming the granular material, the gripper firms up adopting the shape of the enclosed object. Rodriguez and Mason (2012) optimize the shape of non-compliant fingers to yield the same contact point configuration irrespective of object size. Shape adaptability can be enhanced by adding compliant parts and increasing the number of degrees of freedom (Hirose and Umetani, 1978).

An effective way of achieving shape adaptability without increasing the complexity of control is underactuation. The SDM hand (Dollar and Howe, 2010), the Velo gripper (Ciocarlie et al., 2013), the i-HY hand (Odhner et al., 2014), and the Pisa/IIT SoftHand (Catalano et al., 2014) couple the actuation of degrees of freedom using tendon–pulley systems, adapting the shape of the hand to the object while equalizing contact forces.

Shape adaptability can also be accounted for at the perceptual level, when planning grasps. Some works do this by matching hand pre-grasp postures to prototypical geometric shapes (Miller et al., 2003; Eppner and Brock, 2013). Others learn the mapping of hand–object shape match from real data (Lenz et al., 2013). Brost (1986) presents a grasp planner that relies on interactions to reduce state space, but only considers interactions between object and hand.

The nature of hand–object interaction under uncertainty has also been studied in humans. Christopoulos and Schrater (2009) show that humans react to pose uncertainty of an object by aligning the hand with the direction of maximum uncertainty to maximize the probability of establishing force closure at first contact. A similar study concludes that humans maximize the chance of establishing contact with the object, but then perform post-contact postural adaptation (Fu et al., 2013). In this case, the hand acts as a haptic sensor. Both studies show that humans employ contacts with the object to improve the robustness of grasping.

2.3. Interactions between hand, object, and environment

Analogous to the object constraining the motion of the hand, features of the environment may also constrain the motions of hand or object. This is most evident for surfaces which objects rest on, such as tables and floors. These environmental constraints, when used properly, can aid grasping. Furthermore, we postulate that the necessary perceptual information for leveraging such constraints often is easier to obtain than the information required for reliably planning a configuration with force closure property directly.

The idea of environmental constraints appears in early work by Lozano-Pérez, Mason, and Taylor (Lozano-Pérez et al., 1984; Mason, 1985; Erdmann and Mason, 1988). Here, the intrinsic mechanics of the task environment are exploited to eliminate uncertainty and to achieve robustness.

It was not until much later that these concepts gained use in the context of grasping. Recent research leverages environmental constraints in the suggested manner, for example to position the hand relative to the object (Deimel and Brock, 2013), to cage objects (Kazemi et al., 2012; Deimel and Brock, 2013), or to fix an object during planar sliding (Dogar and Srinivasa, 2010; Deimel and Brock, 2013). Furthermore, specialized, simple gripper designs can exploit surface constraints such as floors to reliably pick up a large variety of objects (Xu et al., 2009). The recently introduced concept of extrinsic dexterity (Dafle et al., 2014) shows how the exploitation of environmental constraints can lead to impressive in-hand manipulation capabilities even for simple gripper designs. Also, grasp planning can improve the robustness of grasping by favoring actions that require environmental constraints (Eppner and Brock, 2013).

Some pre-grasp manipulation relies on environmental constraints to improve grasp success. For example, Chang et al. (2008) rotate pan handles into a specific orientation prior to grasping by exploiting the pan’s friction and remote center of mass. This rotation is easy to achieve when the supporting surface is exploited as a constraint for the required motion. Furthermore, non-prehensile manipulation planners also benefit from the consideration of interactions between object and environment (Maeda et al., 2001).

Environmental constraints can also be added to the environment deliberately. In automation and manufacturing, fixtures and part feeders incorporate highly specialized constraints. They are designed to affix a part in space or to move it into a desired orientation. To illustrate, a vibratory bowl feeder uses a set of environmental features in conjunction with a simple transport mechanism (vibration) to achieve complex manipulation behavior. There are approaches that analyze and automatically design the environmental constraints needed to reorient specific objects (Caine, 1994). Though the approach of specializing the environment is economically feasible for mass production, we aim to exploit environmental constraints that are more readily available in a wide range of tasks, settings, object geometries, and perceptual capabilities.

All of the aforementioned methods and mechanisms to exploit environmental constraints rely on multiple compliant interactions involving parts of the environment prior to establishing the final grasp. These phases often are designed to reduce uncertainties in specific variables relevant to grasp success, and may be executed as integrated, swift actions. This blurs the traditional distinction between pre-grasp manipulation and grasping.

We believe that the recent trend towards exploiting environmental constraints and the observation of the same types of behavior in humans represents an opportunity to improve robotic grasping capabilities. To take full advantage of this opportunity, we should understand the strategies humans employ, transfer them to robotic control systems, and also develop robotic hands that facilitate this transfer.

The study of human exploitation of environmental constraints has only received limited attention. For example, Kaneko et al. (2000) extracted a set of grasping strategies from observations of a human subject. These strategies include interactions with environmental constraints. Chang and Pollard (2009) created a taxonomy of human pre-grasp manipulations that employ the support surface by observing video recordings of humans performing object manipulation as part of their daily activities. Many of the documented actions, such as rotating an object on a flat surface, actually rely on the presence of environmental constraints. Wang and MacKenzie (2000) find that the presence of a support surface can increase manipulation speed. The authors attribute this to the support surface’s effect of constraining end-effector motion.

There are also interesting results on the study of human grasping under different kinds of impairments. Severely impairing normal vision of humans with lenses can lead to an almost threefold increase in failed grasps (Melmoth et al., 2009). It has also been shown that tactile impairments (fingertip anesthesia) can lead to ≈ 30% failed grasps even in the presence of a support surface (Gentilucci et al., 1997). Remarkably, both experiments enforced a particular kind of grasp posture. We believe that by imposing constraints on permissible grasp posture, the participants were deprived of the possibility of employing or developing a strategy that counteracts the effects of the impairment.

Kazemi et al. (2014) studied human grasping in a study similar to the one presented in Section 3. They compared surface contact of the hand during grasping in two conditions. In one scenario humans were instructed not to contact the surface during grasping, and in the other they did not receive such instructions and were free to grasp any way they wanted. The experiment revealed that humans are capable of grasping without contacting the support surface when required to do so but in the absence of this constraint exploit environmental constraints extensively. Together with the experiments presented in this paper, it makes a strong case for humans intentionally exploiting the environment to increase the robustness of grasping.

3. Human grasping exploits environmental constraints

In this paper we argue that competent grasping exploits constraints in the environment. In this section, we describe our work towards the identification of successful strategies for the exploitation of environmental constraints in human grasping. In a first step, we define operational measures to quantitatively characterize the exploitation of a specific environmental constraint, namely the support surface of a grasped object. We also show that the interaction with the support surface becomes more pronounced when grasping is made more difficult by impairing human vision. We view this finding as support for our main premise.

3.1. Quantifying contact interactions with support surfaces

We choose the following parameters (also listed in Table 1) to quantify the contact interaction with the support surface during a grasping trial: the number of distinct support contacts, N, the mean travel distance of all support contacts, $\bar{d}$ (spatial extent), the mean duration of all support contacts, $\bar{Δ t_{c}}$ (temporal extent), and the maximum force exerted orthogonal to the support surface, f_max (energetic extent). Additionally, we measure the grasp duration $\bar{Δ t_{g}}$ , that is, the time elapsed between the first contact with either the object or the support surface and object lift. Larger values in these parameters indicate increased interaction with the support surface. We will show that these parameters serve as a meaningful characterization of the interaction with the support surface.

Table 1.

A list of parameters proposed to estimate the extent of interaction during a grasp and which cover different aspects of interaction.

$\bar{d}$	mean travel distance of contacts
$\bar{Δ t_{c}}$	mean duration of contacts
$\bar{Δ t_{g}}$	time from first contact to object lift
f_max	maximum vertical force applied
N	number of distinct contacts

3.2. Experiment

Five right-handed adults (aged 20–25 years, two females) participated in the experiment. They were naive to the rationale behind the experimental design. All participants reported normal or corrected-to-normal vision. The experiment lasted approximately 1.5 hours and each participant received remuneration of €8 per hour.

A grasp trial began with the participant’s hand extended and resting at a start position: see Figure 1. An object was placed at a fixed location on top of a tablet computer located behind an occlusion panel blocking the participant’s view. Then, the occlusion panel was removed and the participant was able to observe the scene. After a delay of 3 s, the participant received an auditory signal to grasp the object. During grasp, the tablet’s touchscreen was used to record the support contact trajectories, from which N, $\bar{Δ t_{c}}$ , and $\bar{d}$ were calculated. The tablet rested on a force/torque sensor used to record contact forces and to estimate f_max . Please refer to Section 3.4 for examples of recorded support contact trajectories and forces. The trial ended with another auditory signal that occurred at lift, in response to which the participant released the object and returned the hand to the start position.

Fig. 1.

Experimental setup of the grasping experiment; top: schematic diagram; bottom: actual setup.

The experiment was performed under two conditions: control and impaired. In the control condition, human vision was not altered. In the impaired condition, the participants wore custom goggles that blurred details of the objects’ shapes and degraded depth perception: see Figure 2. It is difficult to quantify the effect of the goggles, but they allowed us to induce a consistent and severe reduction in human vision. The impaired condition trials preceded the control trials to prevent participants from observing the details of the object shapes. Each participant performed 100 trials: ten objects, five repetitions per object, each under two conditions.

Fig 2.

A goggle with blurring glasses was used to impair vision. The images below show the resulting view of the target objects by the participants in the control condition (left) and the impaired condition (right).

We used the following objects: a button, a salt shaker, a roll of adhesive tape, a matchbox, a marker pen, sunglasses, a comb, a plastic screw, a toy, and a chestnut. All objects were painted black to remove color cues potentially useful for object identification, and to homogenize the contrast with the surroundings (see Figure 2). The tablet’s screen had a white background and was operating at its highest intensity to maximize the contrast between the support surface and the target object. The participants wore a conductive glove to improve the reliability of the touchscreen measurements. The participants were seated as shown in Figure 1, with their head supported by a chin and forehead rest. The setup was adapted to the comfort of the participant and to ensure that the viewing distance to the center of the tablet was ≈ 45 cm. At the beginning of each trial, the touchscreen outlined a bounding box at the center of the tablet’s touchscreen, in which the target object was placed. The experiment was recorded with three cameras that provided frontal, ipsilateral to hand movement, and top views of the grasp movement. Movement onset, that is, when hand velocity exceeded 15 cm/s, was determined using a structured marker attached to the conductive glove and a tracking algorithm that estimated the position of the structured marker. The camera ipsilateral to grasp movement was used to determine when lift occurred, that is, both hand and object were at least 3 mm away from the support surface. The lift detection reliability was ensured by controlled illumination and the high contrast between the black gloved hand and objects, and a white wall that served as background. The force recordings were used to detect the contact time, that is, the first peak in the smoothed force signal after movement onset. The contact time estimation was validated manually.

3.3. Results

The parameters N, $\bar{Δ t_{c}}$ , $\bar{d}$ , f_max , and $\bar{Δ t_{g}}$ measured in the five trials for an object on each condition were averaged for all participants. We performed two kinds of analyses: a correlation analysis and a series of tests for possible effects induced by the impairment.

To check whether the chosen parameters are consistent across subjects and trials within a given condition, we performed a correlation analysis. Figure 3 depicts the Pearson correlation coefficients between all measured indicators, arranged in a cross-correlation matrix for both control and impaired condition. All parameters exhibited strong positive correlations in both conditions, which means that the parameters are consistent and that we can use any subset of the proposed indicators for estimating the extent of interaction. As the parameters cover different aspects of interaction, the strong correlations observed also reduce the chance of a misinterpretation of the results. For example, a participant can exert force on the support surface via the object being grasped, without touching the surface at all. This would potentially make f_max a poor indicator for interaction with the support surface, but as it correlates well with N, $\bar{d}$ , and $\bar{Δ t_{c}}$ we can rule out this alternative explanation in our analysis.

Fig. 3.

All candidate parameters (see Table 1) for estimating grasp difficulty are consistent with each other, as indicated by the Pearson correlation coefficients averaged over participants and trials; top: control condition; bottom: impaired condition.

In the second analysis we tested for an increase of interactions when visually impairing a participant. A set of one-tailed paired t-tests on the five parameters (Holm–Bonferroni corrected with global α = 0.05) revealed a significant effect for each parameter and for all participants. This result is a strong support for our premise stated in Section 1, where a competent grasper will use interactions to counteract uncertainty. The high correlation between grasping time $\bar{Δ t_{g}}$ and the rest of the parameters also suggests that the additional time for the grasp was spent, to some extent, on increasing the interactions with the support surface.

3.4. Examples of grasp trials

We now present some interesting examples of support contact trajectories and contact forces registered during our experiment. The provided data exemplifies the results of the t-test analysis explained earlier on: the participants interacted more with the support surface when visually impaired.

First, we present how participant 3 grasped a button; see Figure 4. The participant slid the button towards the tablet edge before grasping in all 10 trials. However, the slide motion was not generated in the same way in both conditions. In the control condition the participant gently guided the movement of the target with one finger placed on top of the target, and without touching the support surface. In contrast, in the impaired condition support contacts were registered in all trials (see Figure 5 for example images). Interestingly, in trial 5 of the impaired condition, the participant’s middle finger established a support contact while the target was not within the reach of the hand (see Figure 6), and retained it during the slide towards the edge (the arched support contact trajectory on the bottom right). Moreover, the interaction between the hand and the environment was not limited to the top of the tablet. Instead, the participant wrapped his thumb underneath the tablet while performing an edge grasp.

Fig. 4.

Participant 3 grasping the button.

Fig. 5.

Participant 3 grasping a button, trial 1 impaired condition. From top to bottom and from left to right: 1) The fingers establish support contact at the proximal and distal sides of the target. 2) The hand closes upon the object (only the ring finger and the thumb retain support contact; the middle finger is on top of the object). 3) The hand starts sliding the target towards the edge, and the thumb abducts in advance, probably anticipating the arrival of the target. 4) Falls down the edge. 5) The thumb establishes contact with the bottom side of the support surface. 6) Lift complete.

Fig. 6.

Participant 3 initiating support contact when the hand was still reaching for the target in trial 5 of the impaired condition.

Next, we show how participant 2 grasped the same target (see Figure 7). Both conditions had in common that the participant flipped the target by anchoring one finger on one side of the target while pulling from the other side with another finger. Therefore, this participant employed a different strategy than participant 3 to grasp the same object. Interestingly, the fingers that generate support contact trajectories do not necessarily contact the target at any point. For example, in trial 2 of the impaired condition, the flip was performed using only thumb and index fingers but all fingers traveled along the support surface.

Fig. 7.

Participant 2 grasping the button.

Admittedly an object as flat as a button is difficult to grasp without establishing support contact, or without sliding it first towards an edge. However, we also observed support contact on objects that can be grasped directly. An example of this is participant 1 grasping a matchbox (see Figure 8). The support contact trajectories were generated because the participant established several support contacts around the target before closing the hand.

Fig. 8.

Participant 1 grasping the matchbox.

3.5. Discussion

The results of our study support two conclusions. First, the proposed parameters are meaningful for the characterization of the interaction with the support surface, as they exhibit high inter-correlations. Additionally, the high correlations between parameters directly derived from the support contact trajectories (N, $\bar{d}$ and $\bar{Δ t_{c}}$ ), and parameters that are not necessarily directly related to support contact (f_max ) suggest that contact forces are also related to direct interaction with the support surface, at least in grasping experiments involving targets similar ours. This is important because force sensors can be applied in a wider range of situations than touchscreens.

Second, humans increase the interaction with the support surface when their vision is experimentally impaired as indicated by significant differences in the measured parameters between the two conditions. This is consistent with the main premise of this paper, that is, that robust grasping should exploit environmental constraints to compensate for uncertainty. In our experiments, the visual impairment results in an increase in the number of support contacts, and an increase in the duration of support contacts and in their travel distance, and on larger magnitudes of the contact forces.

Grasping time in the impaired condition also increases significantly. Traditionally, this has been interpreted as increased reliance on tactile feedback (Ernst and Banks, 2002; Melmoth et al., 2009). However, since the increase in grasping time correlates with the parameters used to quantify the amount of interaction with the support surface, we also attribute the increased grasping time to increased interaction with the environment. Through observation of the video recordings, we could identify common situations in which the hand interacts with the support surface prior to establishing a grasp, for example when objects are translated or flipped, or whilst the hand closes upon the object. We could also observe support contact when the object was not yet in reach of the hand (see Figure 6), or contact that occurs on the bottom side of the tablet used as support surface. We see these situations as exploitations of the support surface, for example to guide the target during manipulation, to direct the finger trajectories, and to guide the hand trajectory. We also observed that different participants can have different preferences on the strategy to use for a particular situation, which raises the question of what factors drive strategy selection.

In further research we will focus on the systematic identification and detailed study of successful exploitation strategies of environmental constraints, and characterize the conditions for which they are successful. The presented study is a first step towards analyzing human grasp strategies in more detail. We hope to transfer these insights to robots so as to endow them with improved grasping capabilities.

4. Robotic grasping benefits from environmental constraint exploitation

In the previous section, we concluded that humans increase their use of an environmental constraint in response to perceptual uncertainty. In this section, we investigate how robots can exploit such constraints. Our goal is to design grasp strategies that exploit environmental constraints to increase grasp success and to show that there are a variety of environmental constraints that can be leveraged by those strategies.

4.1. Surface-constrained grasp with Barrett hand

We compare two grasp strategies that leverage the same environmental constraint to a different degree. The environmental constraint in this experiment is provided by the supporting table surface. As the height of objects decreases, grasping becomes more difficult. We expect grasp success to be higher if the constraint provided by the table surface to guide finger placement on the object is exploited to a higher degree.

Constant wrist pose: The first strategy was introduced in our prior work (Eppner and Brock, 2013). Grasp poses are generated by fitting geometric primitives like cylinders, spheres, and boxes to depth measurements of the scene. To increase the likelihood of grasp success, pre-grasp poses are refined in response to environmental constraints. For this strategy, the palm of the hand is aligned with the support surface. The hand is then positioned as low as possible above the support surface so that the fingers do not contact the surface during closing. This strategy uses the environmental constraint provided by the support surface to position the hand but does not exploit contact interactions.

Force-compliant closing: The second strategy uses force control to establish contact of the fingertips with the support surface and proceeds to slide the fingers along the surface during closing, maintaining constant contact force by compliantly repositioning the wrist (see Figure 9). Kazemi et al. (2012) present a similar strategy; while they control hand orientation based on force feedback, we employ visual feedback.

Fig. 9.

Force-compliant closing strategy with Barrett hand.

The main difference between the two compared strategies is that the first only attempts to come as close as possible to the surface using RGB-D information about the scene, whereas the second maintains physical contact with the surface throughout the whole grasp. The same environmental constraint, the table surface, is exploited visually in one and haptically in the other.

To evaluate the strategies we placed different-sized cylinders (see Figure 10(a)) on a table in front of a seven-degree-of-freedom whole-arm manipulator equipped with a force-torque sensor and a Barrett Hand BH-262. All experiments reported in this section are averaged over five trials.

Fig. 10.

Objects used in grasping experiments.

Figure 12 shows grasp success as a function of cylinder diameter. While big cylinders could be grasped reliably with both strategies, the grasp of smaller cylinders only succeeded with force-based exploitation of the environmental constraint. The constant-wrist-pose strategy causes the finger tips to hover slightly above the surface when contact with the object is made, due to the circular trajectory during hand closure. This insufficient exploitation of the surface constraint leads to a reduced success rate for small-sized objects. In contrast, the force-compliant finger closing uses the surface constraint at all times to position fingertips as close to the table as possible. Grasp success is not perfect though, as the cylinders can easily roll off the fingertips. An example of this failure mode is shown in Figure 13(a).

This experiment shows that exploiting a surface constraint to a higher degree can lead to more robust grasping.

4.2. Edge grasp with Barrett hand

We want to show that there are multiple environmental constraints that can be exploited. To achieve good grasping performance in a variety of settings and for diverse objects, it is necessary to employ the most appropriate strategy. The multitude of available constraints also necessitates perceptual capabilities to distinguish situations in which one strategy should be preferred over the other. To demonstrate this point, we implemented the slide-to-edge strategy and compared it to the previously presented force-compliant finger closing.

Slide-to-edge: The slide-to-edge strategy exploits a surface and an edge feature in the environment. It contacts the object using the surface, slides it towards an edge, and wraps the thumb around the protruding part of the object to establish a grasp. The different phases of our slide-to-edge strategy are illustrated in Figure 11. This strategy can also be seen as a distinct pre-grasp interaction which reconfigures the object enabling contact on parts of it that were previously inaccessible. A similar strategy was presented in Kappler et al. (2012), focusing on the planning of feasible motions.

Fig. 11.

Slide-to-edge grasp strategy with Barrett hand (see video in Multimedia Extension 1).

We evaluated the slide-to-edge strategy by comparing it to the force-compliant closing strategy for different-sized blocks (see Figure 10(b)) placed on a table as before. For all blocks, the slide-to-edge strategy achieves reliable performance (see Figure 12), whereas the force-compliant strategy is only successful for flat blocks.

Fig. 12.

Comparison of the three grasping strategies.

The slide-to-edge strategy is less sensitive to variation in the size and weight of the blocks. The flat and wide shape of the blocks enables the robot to move parts of them over the edge, creating the opportunity to perform a more reliable grasp on the shorter side of the block. Failure cases for the slide-to-edge strategy included wrong tracking during the visual servoing positioning, missing object contact during sliding, and premature thumb closing.

The force-compliant strategy succeeds when the fingernails jam against one of the block’s sharp edges, as can be seen in Figure 13(b). This is achieved consistently for the smaller blocks. For taller blocks, the fingernails do not contact the object, leading to slip and grasp failure, as seen in Figure 13(c). In a few cases, however, the nails caught the object just before slipping out of the hand. While these cases are counted as grasp success in our experiments, one should note that the intended grasp was not achieved. Success must be attributed to coincidence and the design of the finger nails.

Fig. 13.

Exemplary failure and success cases for the force-compliant closing strategy.

The experiment demonstrates that different ways of exploiting environmental constraints succeed under different conditions. It also shows that the success of exploiting environmental constraints depends on object characteristics in non-trivial ways. It is therefore desirable to employ a variety of grasp strategies for which the conditions of success have been characterized. Perceptual skills then must classify environments according to which of the strategies’ conditions of success are met best.

5. Hands that simplify exploitation of constraints

In this section we present our initial efforts to design hands to simplify exploitation of environmental constraints during grasping. If indeed exploitation of environmental constraints enables robust grasping, such hands should lead to improved grasping performance. Environmental constraints can be exploited most effectively through contact. We therefore design hands so as to attain and maintain contact without the need for sophisticated sensing and control. We achieve this through the extensive use of underactuation, passive compliance, and actuators with low apparent inertia. The initial development goal of the soft hands was to build hands that can grasp objects of uncertain shape using only local, mechanically implementable compliance. In hindsight, that goal is a special case of an environmental constraint: the constraint is the surface of the object being grasped. Many of the design decisions that enable the hand to use the object surface also enable the use of other environmental constraints.

To give an indication on whether soft hands are suitable or even helpful for implementing environmental-constraint-exploiting primitives, we constructed three examples (Figures 20, 21 and 22) using joint control of a seven-degree-of-freedom Mekabot arm with fixed, scripted trajectories and providing compliance by adjusting controller impedances. We evaluated the robustness of the grasps against specific variations of the environment.

5.1. RBO Hand 1

RBO Hand 1 (Deimel and Brock, 2013) is the first design of a very compliant hand and is shown in Figure 14. It employs pneumatic continuum actuators in three fingers and has two deformable pads that form the palm. The hand is highly robust (does not break after thousands of grasps), can withstand blunt collisions, is inherently safe, and easy and cheap to manufacture and repair. This hand achieves robust grasping performance on objects with widely varying geometries, without sensing or control, simply by inflating the continuum actuators (see Figure 16; a more detailed experimental evaluation for these objects can be found in Deimel and Brock, 2013). We obtain these desirable properties at the expense of precise position or force control, of for example the fingertips.

Fig. 14.

RBO Hand 1 consists of a square rectangular plate on which three pairs of PneuFlex continuum actuators are mounted as fingers at a 30° angle. Opposing the fingers, a simple cylindrical pad is mounted, which is made from a sheet of rubber. The intermediate section is also padded with rubber.

5.1.1. Surface-constrained grasp

The first environmental-constraint-exploiting grasp implemented on RBO Hand 1 was the surface-constrained grasp (Deimel and Brock, 2013). Its steps and execution are illustrated in Figure 20, and for a particularly difficult object in Figure 20(b). The strategy makes extensive use of environmental constraints. It uses contact between the palm and the support to level the hand with the object. The fingers slide along the support to establish reliable contact with the object. Finally, the fingers adapt to the shape of the object to establish a robust grasp. These ways of exploiting environmental constraints are facilitated by the hand’s design and do not require sensing or control.

5.1.2. Edge grasp

We also implemented the slide-to-edge grasp from Section 4.2 for RBO Hand 1, but in a simpler version, omitting the sliding step. Its steps and execution are illustrated in Figure 21. In the first phase, the hand’s palm establishes contact with the edge, eliminating position uncertainty. Subsequently, the fingers are flexed and the fingertips establish contact with the table, achieving caging. The hand rotates about the edge/palm contact to ensure contact between the fingers and the support surface, while the compliant fingers slide along the support surface until a grasp is established. Finally, the hand retracts from the edge at an angle of 15^°, lifting the fingertips from the surface and detaching the palm from the edge at the same time.

5.2. RBO Hand 2

The latest iteration of hand design is a prototype of an anthropomorphic hand (Deimel and Brock, 2014), shown in Figure 15. It has seven individual PneuFlex continuum actuators, one for each finger, and two curved ones making up the palm. The palm and fingers of the hand are mounted on a flexible, printed scaffold, which augments their compliance and lowers forces on impact. The actuated palm results in a dexterous thumb, but also provides a compliant pad to grasp against. The scaffold is stabilized by flexible connections between fingers and palm. RBO Hand 2 shares the same actuator technology as RBO Hand 1, but its fingers are designed to be approximately four times stronger and have a linearly decreasing impedance instead of a constant one along the fingers. The hand is capable of enacting 31 out of 33 grasps of the Feix grasp taxonomy using only four actuation signals by relying on its mechanical compliance (Deimel and Brock, 2014).

Fig. 15.

RBO Hand 2 consists of a flexible polyamide scaffold on which four fingers and a palm–thumb compound are mounted. Fingers and palm are made of PneuFlex continuum actuators. On its backside splitters are attached to distribute air from two actuation channels to the individual actuators.

A big advantage of both hand designs is that the most exposed parts contain no rigid components able to concentrate forces. It is therefore very safe. Errors usually do not lead to catastrophic failure as fingers and palm can comply in every direction. The low inertia of the PneuFlex continuum actuators also facilitates fast collisions without excessive, damaging contact pressures. Additionally, the low actuator impedance and fast response on disturbances help to maintain contact with surfaces during hand motion. These properties greatly simplify the implementation of environmental-constraint-exploiting primitives.

5.2.1. Slide-to-wall grasp

For RBO Hand 2 we implemented a strategy that uses walls: a constraint that can be found as part of bowls, drawers, shelves and boxes. The strategy’s steps and execution are illustrated in Figure 22. The slide-to-wall grasp exploits the corner created by two surfaces in addition to the two surfaces themselves. In the slide phase, the robot lowers the wrist until it touches the table. It drags its fingers across the surface to slide the object into the corner to finish the slide phase. The object is now caged from four sides by table, fingers, wall, and gravity. We then reorient the hand by first unloading the fingers (backward motion), and rotating approximately around the fingertips. As the motions are executed using joint space interpolation, the fingers may compensate for resulting positioning errors with bending. Then, the hand is moved compliantly against the wall to slip the fingers under the object, which is constrained in horizontal motion by the wall. This phase effectively replaces the table constraint with the fingers. Then, the fingers and palm are inflated slightly (approximately 15% of final actuator pressure), and the hand is rotated to create a cage with the wall. The fingers are fully flexed to grasp the object. This last step is similar to the surface-constrained grasp shown in Figure 20(a), but with gravity being oriented differently.

5.3. Robustness under uncertainty

Exploitation of environmental constraints should lead to successful grasps in a broad range of situations. By ‘outsourcing’ the interaction into hardware with accompanying motion primitives, the robot does not need to perceptually distinguish between situations where the same action yields the same outcome. Therefore, robustness and predictability of environmental-constraint-exploiting primitives against variations also directly simplify perception and planning.

To evaluate robustness of the hand designs and accompanying grasping strategies, we mapped grasp success against several grasp-relevant parameters: object shape, object size, object placement, and environmental constraint placement.

Object shape: In previous work, we demonstrated the ability of RBO Hand 1 to grasp a diverse set of objects of comparable size (see Figure 16) (Deimel and Brock, 2013). Here the many compliant degrees of freedom of the hand are used to adapt to the surface shape of the object. Figure 16 also shows that RBO Hand 1 is able to grasp deformable objects often considered hard to grasp, such as tissue or a water balloon. The intrinsic compliance can adapt to shape changes without the need for continuous shape perception. In accordance with these initial findings, RBO Hand 2 was also shown to be able to grasp many differently shaped objects (Deimel and Brock, 2014). Compliant actuation has also been successfully used by the SDM hand (Dollar and Howe, 2010) and the positive pressure gripper (Amend et al., 2012) to automatically adapt to diverse object shapes.

Fig. 16.

Different objects that can be grasped with RBO Hand 1.

Object placement: In two experiments, we measured robustness of the strategies when objects are displaced from their expected position. For both experiments we used the set of cylinders shown in Figure 10(a). The set of blocks from Figure 10(b) used in the experiments with the Barrett hand in Section 4 cannot be grasped by RBO Hand 1 due to limitations in actuation and hand aperture. Objects were displaced along one axis in 12 (10) 20 mm increments, using nine different cylinder sizes, for a total of 108 (90) trials for the surface-constrained (slide-to-edge) grasp. To create a dense spatial coverage with a feasible number of experiments, every combination of object placement and size was sampled only once.

The results of these experiments are shown in Figures 17 and 18. Both strategies achieved grasp success in large and contiguous areas of the explored parameter space. For graspable objects, displacements can vary in large ranges due to the exploitation of environmental constraints in the various steps. Consistent grasp success under significant variations in object placement is a strong indication for the robustness of constraint exploitation facilitated by the hand design. Note that the hand does not use sensing or control to achieve this grasping performance.

Fig. 17.

Surface-constrained grasp. Distance measure as indicated in Figure 20(a). Circles represent successful grasps.

Fig. 18.

Slide-to-edge grasp. Distance measured horizontally from lower edge of palm plate to closest object surface. Circles indicate successful grasps.

Object size: The tolerance of the two tested grasps to changes in cylinder diameter can also be extracted from Figure 17 and Figure 18. Objects larger than 75 mm cannot be grasped, as the hand’s fingers are not able to reach around far enough to create force closure. For small objects, the edge grasp is superior to the surface-constrained grasp, which only works for cylinder diameters above 16 mm. As with object placement, object size can vary considerably without affecting grasp success, making the grasps robust and predictable.

The results in Figures 17 and 18 also show that different grasps are successful under different conditions. The surface-constrained grasp requires cylinders to be at least 22 mm in diameter, whereas the edge grasp requires the presence of an edge within about 100 mm of the object. This confirms the results from Section 4.2 and emphasizes the necessity of employing multiple strategies in response to the specific grasp problem.

Constraint placement: To be able to robustly execute a constraint-exploiting action, for the environmental-constraint-exploiting primitive it is also necessary to tolerate uncertainties in the placement of environmental constraints themselves. In the third experiment, we evaluated the influence of wall direction on the slide-to-wall grasp by varying the angle between the two surfaces exploited as environmental constraints. While most walls are vertical, some are not, such as the walls of some bowls and boxes. The grasp sequence was initially constructed using a wall at a 60^° angle. During the experiment, the wall angle was changed from α = 40^° to α = 90^°, in 10^° increments and tested 10 times. After that, the two interesting border regions were identified and two additional angles were tested to increase resolution for a total of 80 grasps.

The results are shown in Figure 19. The grasp could be successfully executed without any adaptation of the actuation, in a large range of wall orientations, from approximately 45^° to 90^°. Larger angles could not be tested, because the wrist collided with the wall constraint during the slide motion. Larger angles would have increased the deflection and result in a larger force by the joint controllers. To avoid damage to the arm, angles larger than α = 90^° were not tested and should be considered unsuccessful. Even then, the grasp can tolerate large changes in the orientation between the two required surfaces, which in turn lowers the difficulty of sensing the presence of the required environmental constraints.

Fig. 19.

Success of slide-to-wall grasp under varying wall angles relative to a horizontal table surface.

Fig. 20.

Surface-constrained grasp with RBO Hand 1 (see video in Multimedia Extension 1).

Fig. 21.

Edge-grasp strategy with RBO Hand 1 (see video in Multimedia Extension 1).

Fig. 22.

Slide-to-wall grasp strategy with RBO Hand 2 (see video in Multimedia Extension 1).

6 Conclusion

The work presented in this paper describes the early stages of an integrated research agenda in robotic grasping. This agenda combines the study of human grasping to identify strategies and principles leading to their competencies with the transfer of these principles to robotic grasp planners as well as to robotic hand design.

Informed by a growing body of research in robotic grasping, we formulated the premise that robust and reliable grasping must exploit environmental constraints during the grasping process. In support of this premise, we presented experiments showing that humans respond to increased difficulty in the grasping problem by increasing the exploitation of environmental constraints. We believe that the study of human exploitation strategies will provide important insights into how robotic grasping algorithms can achieve robust grasping performance.

Following these insights, we presented several such strategies on three different robot platforms. Each of the strategies was tailored to exploit constraints commonly present in real-world grasping scenarios. We demonstrated the success of constraint exploitation in real-world grasping experiments.

Finally, we demonstrated the utility of designing hands to facilitate the exploitation of environmental constraints by presenting two types of mechanically compliant and highly deformable hands. Both hands robustly grasp objects of varying sizes and shapes, without the need for explicit force sensing or feedback control, and make collision and interaction with the environment simple to implement.

Viewed collectively, the experimental results on human and robotic grasping presented in this paper provide strong support for the view that the ability to exploit environmental constraints is a crucial component in the development of competent robotic grasping and manipulation systems.

Footnotes

Appendix: Index to Multimedia Extension

Archives of IJRR multimedia extensions published prior to 2014 can be found at http://www.ijrr.org, after 2014 all videos are available on the IJRR YouTube channel at http://www.youtube.com/user/ijrrmultimedia

SAGE-Journals-Accessible-Video-Player 10.1177/0278364914559753.M1 0278364914559753_video1 Extension 1

Demonstrations of environmental constraint exploiting grasping strategies

Funding

We gratefully acknowledge the funding provided by the Ministry of Education and Research (BMBF), awarded by the Alexander von Humboldt foundation.

References

Amend

Brown

Rodenberg

. (2012) A positive pressure universal gripper based on the jamming of granular material. IEEE Transactions on Robotics 28(2): 341–350.

Balasubramanian

Brook

. (2012) Physical human interactive guidance: Identifying grasping principles from human-planned grasps. IEEE Transactions on Robotics 28(4): 899–910.

Brost

(1986) Automatic grasp planning in the presence of uncertainty. In: IEEE international conference on robotics and automation (ICRA), pp. 1575–1581.

Caine

(1994) The design of shape interactions using motion constraints. In: IEEE international conference on robotics and automation (ICRA), pp. 366–371.

Catalano

Grioli

Farnioli

. (2014) Adaptive synergies for the design and control of the Pisa/IIT SoftHand. The International Journal of Robotics Research 33(9): 768–782.

Chang

Pollard

(2009) Video survey of pre-grasp interactions in natural hand activities. In: Robotics: Science and systems (RSS) 2009 workshop on understanding the human hand for advancing robotic manipulation.

Chang

Zeglin

Pollard

(2008) Preparatory object rotation as a human-inspired grasping strategy. In: IEEE-RAS international conference on humanoids, pp. 527–534.

Christopoulos

Schrater

(2009) Grasping objects with environmentally induced position uncertainty. PLoS Computational Biology 5(10): e1000538.

Ciocarlie

Mier Hicks

Stanford

(2013) Kinetic and dimensional optimization for a tendon-driven gripper. In: IEEE international conference on robotics and automation (ICRA), pp. 217–224.

10.

Ciocarlie

Allen

(2009) Hand posture subspaces for dexterous robotic grasping. The International Journal of Robotics Research (IJRR) 28(7): 851–867.

11.

Controzzi

Cipriani

Carozza

(2014) Design of artificial hands: A review. In: Balasubramanian

Santos

(eds) The Human Hand as an Inspiration for Robot Hand Development (Springer Tracts in Advanced Robotics, vol. 95). New York, NY: Springer, pp. 219–247.

12.

Cutkosky

(1989) On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Transactions on Robotics and Automation 5(3): 269–279.

13.

Dafle

Rodriguez

Paolini

. (2014) Extrinsic dexterity: In-hand manipulation with external forces. In: IEEE international conference on robotics and automation (ICRA), 31 May–7 June, pp. 1578–1585.

14.

Deimel

Brock

(2013) A compliant hand based on a novel pneumatic actuator. In: IEEE international conference on robotics and automation (ICRA), Karlsruhe, Germany, pp. 472–480.

15.

Deimel

Brock

(2014) A novel type of compliant, underactuated robotic hand for dexterous grasping. In: Robotics: Science and systems (RSS), Berkeley, USA.

16.

Deimel

Eppner

Álvarez Ruiz

. (2013) Exploitation of environmental constraints in human and robotic grasping. In: International symposium on robotics research (ISRR), Singapore.

17.

Dogar

Srinivasa

(2010) Push-grasping with dexterous hands: Mechanics and a method. In: IEEE/RSJ international conference on intelligent robots and systems, pp. 2123–2130.

18.

Dollar

Howe

(2010) The highly adaptive SDM hand: Design and performance evaluation. The International Journal of Robotics Research (IJRR) 29(5): 585–597.

19.

Eppner

Brock

(2013) Grasping unknown objects by exploiting shape adaptability and environmental constraints. In: IEEE international conference on intelligent robots and systems (IROS).

20.

Erdmann

Mason

(1988) An exploration of sensorless manipulation. The International Journal of Robotics Research (IJRR) 4(4): 369–379.

21.

Ernst

Banks

(2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870): 429–433.

22.

Feix

Pawlik

Schmiedmayer

. (2009) A comprehensive grasp taxonomy. In: Robotics: Science and systems (RSS).

23.

Ushani

Jentoft

. (2013) Human reach-to-grasp compensation with object pose uncertainty. In: 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6893–6896.

24.

Gentilucci

Toni

Daprati

. (1997) Tactile input of the hand and the control of reaching to grasp movements. Experimental Brain Research 114(1): 130–137.

25.

Grebenstein

Chalon

Friedl

. (2012) The hand of the DLR hand arm system: Designed for interaction. The International Journal of Robotics Research (IJRR) 31(13): 1531–1555.

26.

Hirose

Umetani

(1978) The development of soft gripper for the versatile robot hand. Mechanism and Machine Theory 13(3): 351–359.

27.

Kaneko

Shirai

Tsuji

(2000) Scale-dependent grasp. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 30(6): 806–816.

28.

Kappler

Chang

Pollard

. (2012) Templates for pre-grasp sliding interactions. Robotics and Autonomous Systems 60(3): 411–423.

29.

Kawasaki

Komatsu

Uchiyama

. (1999) Dexterous anthropomorphic robot hand with distributed tactile sensor: Gifu Hand II. In: IEEE international conference on systems, man, and cybernetics, pp. 782–787.

30.

Kazemi

Valois

Bagnell

JAD

. (2012) Robust object grasping using force compliant motion primitives. Technical Report CMU-RI-TR-12-04, The Robotics Institute, Carnegie Mellon University, PA.

31.

Kazemi

Valois

Bagnell

. (2014) Human-inspired force compliant grasping primitives. Autonomous Robots 37(2): 209–225.

32.

Lenz

Lee

Saxena

(2013) Deep learning for detecting robotic grasps. In: Robotics: Science and systems (RSS), Berlin, Germany.

33.

Lozano-Pérez

Mason

Taylor

(1984) Automatic synthesis of fine-motion strategies for robots. The International Journal of Robotics Research (IJRR) 3(1): 3–24.

34.

Maeda

Kijimoto

Aiyama

. (2001) Planning of graspless manipulation by multiple robot fingers. In: IEEE international conference on robotics and automation (ICRA), pp. 2474–2479.

35.

Mason

(1985) The mechanics of manipulation. In: IEEE international conference on robotics and automation, pp. 544–548.

36.

Mason

(2001) Mechanics of Robotic Manipulation. Cambridge, MA: MIT Press.

37.

Melmoth

Finlay

Morgan

. (2009) Grasping deficits and adaptations in adults with stereo vision losses. Investigative Ophthalmology & Visual Science 50(8): 3711–3720.

38.

Miller

Allen

(2004) Graspit! A versatile simulator for robotic grasping. IEEE Robotics and Automation Magazine 11(4): 110–122.

39.

Miller

Knoop

Christensen

. (2003) Automatic grasp planning using shape primitives. In: IEEE international conference on robotics and automation (ICRA).

40.

Odhner

Jentoft

Claffee

. (2014) A compliant, underactuated hand for robust manipulation. The International Journal of Robotics Research 33(5): 736–752.

41.

Prattichizzo

Trinkle

(2008) Grasping. In: Siciliano

Khatib

(eds.) Springer Handbook of Robotics. Berlin: Springer, pp. 671–700.

42.

Rodriguez

Mason

(2012) Grasp invariance. The International Journal of Robotics Research (IJRR) 31(2): 236–248.

43.

Santello

Flanders

Soechting

(1998) Postural hand synergies for tool use. The Journal of Neuroscience 18(23): 10,105–10,115.

44.

Wang

MacKenzie

(2000) The role of contextual haptic and visual constraints on object manipulation in virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 532–539.

45.

Deyle

Kemp

(2009) 1000 trials: An empirically validated end effector that robustly grasps objects from the floor. In: IEEE international conference on robotics and automation (ICRA), pp. 2160–2167.