Toward Damage-Less Robotic Fragile Fruit Grasping: A Closed-Loop Force Control Method for Pneumatic-Driven Soft Gripper

Abstract

Fragile fruit uploading and packaging are labor-intensive and time-consuming steps in postharvest industry. With the aging of the global population, it is supposed to develop robotic grasping systems to replace manual labor. However, damage-less grasping of fragile fruit is the key problem in robotization. Inappropriate grasping force will result in damage, early-stage bruise, or slip. Benefits from the advantages of softness and compliance of a pneumatic-driven soft gripper have been widely adopted for agricultural product and food manipulation. Nevertheless, pneumatic gripper is a complex, multivariable, nonlinear, and long time-delay control system, which is difficult to achieve robust closed-loop grasping force control. In this study, we aim to solve this problem and developed a robotic grasping force control system with pneumatic gripper and matrix force sensor. The force distribution condition was explored to tackle the problem in changing of the main contact point. A double closed-loop control method was proposed based on Kalman filter (KF) and proportion integration differentiation controller with dead band. The external and internal control loops were force controller and air pressure of the pump controller, respectively. The double closed-loop controller with dead band achieved robust grasping force control through air pressure. The experimental results validated the effectiveness of the KF method for denoising and the matrix force visualization method for exploring grasping mechanism. Ablation studies were carried out to demonstrate the effectiveness of the multiple grasping force sensing units in matrix form and the dead band in the controller. The maximum steady-state error was 0.07 N. In addition, the generalization performance and the antidisturbance ability of the grasping force control system was also validated. In summary, the problem in closed-loop control of the grasping force for pneumatic gripper has been solved in our study, and the method in this research is potential to be deployed in fruit postharvest industry.

Introduction

With the growing of the annual yield of fruit production, postharvest commercialization has become a large industry all over the world.¹ In general, the following processes are necessary in the postharvest industry, including uploading, washing, quality detection, grading, and packaging.² Until now, the automation has largely been achieved for quality detection and grading in most companies,³ such as REEMOON, GREEFA, Robotics Plus, and MAF RODA. Among all the postharvest processes, uploading and packaging are the most time-consuming and labor-intensive steps, which need continuous and manual pick-and-place manipulation of the fruit.^2,4 With the aging of global population, the problem of labor shortage is becoming increasingly prominent.^5,6 Robotic grasping and manipulation⁷ are supposed to replace the manual labor and improve the automation and intelligence level of the postharvest industry.^8–11

However, different from industrial workpieces, due to the diversity in sizes, shapes, colors, weights, firmness, and ripeness for agricultural products, robotic uploading and packaging are still difficult to achieve.^12–15 In recent years, a lot of studies have been carried out and a lot of large-scale robotic grasping datasets have been made public,¹⁶ such as DexNet,¹⁷ GraspNet,¹⁸ Conell,¹⁹ and YCB.²⁰ Fragile fruits, such as peach, apple, and orange, were included in these datasets. Nevertheless, the above studies mainly focus on visual perception and grasp pose estimation in robotic grasping. The grasping objects were models made of plastic, the weights light, and the diversities in firmness and ripeness were not taken into consideration.^21–24 For real fruit with various firmness and ripeness, inappropriate grasping force is potential to cause damage, early-stage bruise, or slip.²⁵ In postharvest industry, these situations would influence the added value and the shelf life, thus causing serious economic losses. In general, for fragile fruit, robust grasping force control and damage-less grasping strategies are key problems that deserve to be studied.^26,27

To solve this problem in robotic grasping, researchers have carried out many studies toward fragile fruit. Friedl et al.²⁸ developed an electric rigid gripper called CLASH, and evaluated tactile sensors for fruit grasping.²⁹ Although an electric rigid gripper has the advantages of high control precision and fast response speed, the softness and compliance are insufficient. Damage or early-stage bruise may occur among fragile fruits. Pneumatic-driven soft grippers^30,31 have attracted considerable attention of researchers for their better softness and compliance, low cost, and ease of fabrication,³² and have been widely used for robotic grasping of agricultural products and food.²⁵ However, the pneumatic soft gripper is a complex, multivariable, nonlinear, and long time-delay control system, and it is difficult to set up its mathematical model to achieve state estimation or closed-loop control of bending angle or grasping force. To solve this problem, Ye et al.³³ proposed a bending angle modeling method for soft fiber-reinforced pneumatic actuator based on machine learning algorithm trained from finite element data, and achieved closed-loop control of the bending angle. While for fragile fruits with various sizes and shapes, grasping force is still another difficult factor to achieve closed-loop control on pneumatic gripper. Hence, other studies developed pneumatic grasping systems equipped with a force sensor, and achieved grasping force measurement,^34,35 grasping patterns of human hands analysis,^36–38 and varieties or firmness recognition.^39,40 For example, Li and Liu⁴¹ developed a bionic adaptive end-effector equipped with force and posture sensors to achieve pear harvesting. However, the grasping force control was open loop and judged by force threshold. The grasping force was unstable when the system is disturbed. Gao et al.⁴² developed a pneumatic gripper for cherry tomato harvesting in greenhouse. However, the measured grasping force of human hands was not applied for pneumatic gripper control. There were still gaps between manual and robotic grasping. In summary, the closed-loop control of the grasping force for pneumatic-driven soft gripper is still a problem to be solved.⁴³

Moreover, matrix force sensor has also attracted considerable attention of researchers.^27,44 Compared with a force sensor with single sensing unit, the matrix force sensor has the advantage of higher spatial and temporal resolution, and could obtain the distribution condition of the grasping force.²⁶ In related studies for agricultural products and food grasping and manipulation,^35,36,38,45 force sensors with a single sensing unit were applied for grasping force analysis and control. For example, Yang et al.⁴⁶ developed a liquid lens-based optical tactile sensing unit that contains a self-sensing fingertip and a liquid lens module. An online learning self-tuning nonlinearity impedance controller was proposed for grasping force control of the soft gripper. While the sensing unit in the tactile sensor was single and the force distribution condition could not be obtained. As is known to all, the surface smoothness and consistency are different for the agricultural products. The contact point and condition between objects and force sensors or grippers may be influenced. Besides, the changes in air pressure and bending angle will result in the changes in main contact point. Although closed-loop control of the grasping force for the pneumatic actuator has been achieved in some related studies,^47,48 the contact force was applied on a load cell or a single point. The contact condition was idealized and simplified, and the complex changes in force distribution were not taken into consideration. Therefore, the robustness for force sensors with a single sensing unit is difficult to ensure.

In this research, we aim to address the issues mentioned above. Specifically speaking, the pneumatic-driven soft gripper is a multivariable, nonlinear, and long time-delay controlled system. To achieve closed-loop control of the grasping force, the hardware system needs to be specifically designed and a robust control method needs to be developed to obtain better transient state and steady-state performances. Overall, the main contributions of this article are summarized and listed as follows:

To achieve damage-less robotic fragile fruit grasping for postharvest industry, a robotic grasping force control system based on pneumatic-driven soft gripper equipped with matrix force sensor was developed.

A grasping force control method for robotic pneumatic gripper was proposed based on Kalman filter (KF) and double proportion integration differentiation (PID) controller with dead band.

Extensive experiments were carried out to validate the effectiveness and generalization performance in grasping force control of the proposed method. Both the transient and steady-state performances were improved.

Materials and Methods

In this section, we introduce the robotic fragile fruit grasping system developed in this study and the proposed grasping force control method based on KF and double PID controller with dead band. In addition, some evaluation metrics are also introduced for experiments.

Materials

It is well-known that a human can achieve damage-less fragile fruit grasping without difficulty based on their ability to force sensing, feedback, and control. Inspired by human grasping behavior illustrated in Figure 1A, we developed a robotic grasping system based on pneumatic soft gripper and matrix force sensor, as shown in Figure 1B. The detailed features and corresponding parameters of the main devices adopted in this research are listed in Table 1 for demonstration.

FIG. 1.

The application scenario, the developed robotic peach grasping system, and the proposed grasping force control method. (A) The robotic peach grasping system is supposed to be deployed for uploading and packaging in fruit postharvest industry to replace monotonous manual manipulation. (B) The developed robotic peach grasping system in this study mainly consists of two-finger pneumatic soft gripper, 4 × 4 matrix force sensor, air pump, robotic manipulator, and computer. (C) The proposed grasping force control method for pneumatic soft gripper was based on Kalman filter (KF) and double proportion integration differentiation (PID) controller with dead band.

Table 1.

The Detailed Features and Corresponding Parameters of the Main Devices Adopted in This Study

Device	Feature	Parameter
Matrix force sensor	Range (N)	0–5
	Number of sensing units	4 × 4 = 16
	Sensing area (mm)	10 × 10
	Thickness (mm)	0.15
ADC	Data bits (bit)	16
Pneumatic gripper	Maximum pressure (kPa)	300
	Maximum vacuum (kPa)	−80
	Space (mm)	70
	Load (g)	400
	Stroke (mm)	435
	Width (mm)	24
	Length (mm)	97
Air pump	Maximum pressure (kPa)	300
	Maximum vacuum (kPa)	−80
	Control mode	MODBUS RTU 485

ADC, analog-to-digital converter.

First of all, a robotic manipulator Franka Panda (FRANKA EMIKA GmbH, Munich, Germany) with 7 degrees-of-freedom was adopted for pick-and-place manipulation. A two-finger pneumatic soft gripper F-A5T/LF1[H] (Soft Touch Robot Technology Co., LTD, Suzhou, China) was attached to the end of the robotic manipulator, the space of the left and right fingers was adjustable to fit different types of fruit with various diameters. The gripper was controlled by a programmable controlled air pump ACU2-H (Soft Touch Robot Technology Co., LTD, Suzhou, China). The communication protocol for air pressure control was MODBUS RTU 485. To achieve force sensing and control, two matrix force sensors RX-M0404S (Roxi Electronic Technology Co., LTD, Changzhou, China) were installed at the end of the left and right grippers, respectively. The measuring range was 5 N and the number of the sensing unit was 4 rows and 4 columns. Two 16-bit analog-to-digital converters (ADCs) were attached to receive the grasping force value.

Furthermore, a binocular stereo depth camera ZED Mini (Stereolabs Inc., San Francisco, USA) was installed at the top of the system for depth perception and grasp pose estimation. The grasp pose sampling method based on force closure principle was proposed to estimate the proper grasp pose from the obtained depth image. This pose was transformed into the robot base coordinate system to execute grasping task and avoid slip phenomenon during grasping. The main devices in the system mentioned above were controlled by an S3 Pro computer (MECHREVO Corporation, Beijing, China) with Intel Core i5-11300H CPU, 16 GB memory (3200 MHz), and 512 GB hard disk. All the control software were run on the Ubuntu 20.04 Linux operating system and integrated with ROS Noetic Ninjemys.

Principle, calibration, and KF of the matrix force sensor

The matrix thin film force sensor adopted in this research was based on the piezoresistive principle. The output resistance of each sensing unit is inversely proportional to the applied force. This relationship is represented as the following equation: $F \propto \frac{1}{R},$ (1)where $R = [R_{1}, R_{2}, \dots, R_{16}]$ is the resistance of each sensing unit, and $F = [F_{1}, F_{2}, \dots, F_{16}]$ is the grasping force applied on each sensing unit, respectively. For ease of measurement, the resistance was converted into voltage through the designed circuit: $U = V_{c c} \times (1 + \frac{R_{M}}{R}),$ (2)where $U = [U_{1}, U_{2}, \dots, U_{16}]$ is the output voltage of each sensing unit. $V_{c c}$ denotes the no-load voltage and $R_{M}$ represents the feedback resistance. Based on the above principles, there is a linear relationship between the force $F$ and the output voltage $U$ , which is represented as the following equation: $F = a \times {ADC}_{16} (U) + b,$ (3)where ${ADC}_{16} (\cdot)$ denotes the output of the ADC module. Through the ADC module, the analog voltage is converted into digital voltage. Sensor calibration step was carried out on a texture analyzer (TA-XT2i, Stable Micro Systems LTD, Surrey, UK) to determine the gain $a$ and the bias $b$ in Eq. (3). The calibration results were as follows: ${\begin{matrix} L eft Finger : F = \frac{5}{3366} \times {ADC}_{16} (U) - \frac{1000}{561} \\ R ight Finger : F = \frac{5}{3166} \times {ADC}_{16} (U) - \frac{3500}{1583} \end{matrix} .$ (4)

Given the measured average grasping force $\bar{F} (k) \in R^{1 \times 1}$ at moment $k$ , the KF step is supposed to filter the noise component in the signal and output the predicted average grasping force $\hat{F} (k) \in R^{1 \times 1}$ at moment $k$ . The KF algorithm consists of five steps. First, the prior state ${\hat{F}}^{-} (k)$ for the next time point $k$ is calculated through the following equation: ${\hat{F}}^{-} (k) = A \hat{F} (k - 1),$ (5)where $A$ denotes the state transition matrix, and $\hat{F} (k - 1)$ is the predicted grasping force for the last time point $k - 1$ . In addition, the prior state error covariance matrix $P^{-} (k)$ is also calculated, $P^{-} (k) = AP (k - 1) A^{T} + Q,$ (6)where $P (k - 1)$ is the error covariance matrix for the last time point $k - 1$ , and $Q$ represents the process noise covariance matrix of the system. From another perspective, the process noise covariance matrix represents the confidence in the predicted value. A smaller value will result in smoother predicted curve. Next, the Kalman gain $K (k)$ is updated: $K (k) = P^{-} (k) H^{T} {(H P^{-} (k) H^{T} + R)}^{- 1},$ (7)where $H$ and $R$ denote the measurement matrix and the measurement noise covariance matrix of the system, respectively. From another perspective, these two matrices represent the confidence in the measured value from the force sensor. A smaller value will result in closer with the measured grasping force. Then, the predicted grasping force for the next time point $\hat{F} (k)$ can be calculated through the next equation, $\hat{F} (k) = {\hat{F}}^{-} (k) + K (k) (\bar{F} (k) - H {\hat{F}}^{-} (k)) .$ (8)

Finally, the error covariance matrix $P (k)$ is updated to the next time point $k$ through the following equation: $P (k) = P^{-} (k) - K (k) H P^{-} (k) .$ (9)

The above processes illustrated in Eqs. (5)–(9) are circulated during the force acquisition process.

Closed-loop control of the grasping force

In this research, we propose double PID controller with dead band to achieve grasping force control. The control objective is the grasping force and the control variable is the air pressure. On the one hand, the external controller (force controller) calculates the error in grasping force and compensates the error. On the other hand, the internal controller (air pressure controller) calculates the error in output air pressure from the air pump and compensates the error. In the following content, we introduce the external and internal controllers, respectively.

The surface of the fruit is not perfectly flat and there is great diversity in the shapes of fruit. For these reasons, the grasping force on single sensing unit is unstable and unreliable. Since we adopted 4 × 4 matrix force sensor, the average grasping force from 16 sensing units was used for the subsequent processing. For external controller represented in blue color in Figure 1C, the average grasping force at time point $k$ is calculated in the first place through the following equation: $\bar{F} (k) = \frac{1}{N} \sum_{n = 1}^{N} F_{n} (k),$ (10)where $n$ is the channel number of the sensing unit, and $N = 16$ represents the number of the sensing unit for one matrix force sensor. The KF process is executed to obtain the filtered average grasping force $\hat{F} (k)$ . The error between the filtered average grasping force and the desired grasping force $e_{f} (k)$ is calculated through the next equation: $e_{f} (k) = F_{d} - \hat{F} (k),$ (11)where $F_{d}$ denotes the desired grasping force. Then, the incremental PID controller with dead band is designed for compensation quantity $Δ p_{d} (k)$ calculation as follows: $Δ p_{d} (k) = K_{P}^{f} [e_{f} (k) - e_{f} (k - 1)] + K_{I}^{f} e_{f} (k) + K_{D}^{f} [e_{f} (k) - 2 e_{f} (k - 1) + e_{f} (k - 2)],$ (12)where $K_{P}^{f}$ , $K_{I}^{f}$ , and $K_{D}^{f}$ denote the parameters for proportion, integration, and differentiation in the external force controller, respectively. The proportional and differential terms are introduced to improve the transient response, and the integral term is introduced to suppress the steady-state error. Expression $e_{f} (k) - e_{f} (k - 1)$ represents the increment of the error of the grasping force. It is worth noting that, the dead band is appended to overcome the long time-delay problem in pneumatic gripper control. The dead band could prevent the system from frequent fluctuation around the desired grasping force, and avoid the switch of the air pump from frequent starts and stops. The dead band is represented as the following equation: $e_{f} (k) = {\begin{matrix} 0, i f | e_{f} (k) | \leq E_{0} \\ e_{f} (k), i f |e_{f} (k)| > E_{0} \end{matrix},$ (13) where $E_{0}$ represents the range of the dead band. Experimental results demonstrated that the performance of the control system becomes best when $E_{0} = 1.5 N$ .

Owing to the nonlinear and long time-delay properties, it is difficult to achieve satisfactory results with merely external control loop. Therefore, we designed the air pressure controller as the internal control loop. For internal controller represented in green color in Figure 1C, the set value for air pressure at time point $k$ is calculated in the first place as follows: $p_{d} (k) = p_{d} (0) + \sum_{0}^{k} Δ p_{d} (k),$ (14)where $p_{d} (0)$ is the initial set value for the air pump. $Δ p_{d} (k)$ is the compensation quantity of the air pressure at time point $k$ , which could be calculated from the external controller illustrated in Eq. (12). The error in air pressure $e_{p} (k)$ is calculated in the next step: $e_{p} (k) = p_{d} (k) - p (k),$ (15)where $p (k)$ is the measured air pressure feedback from the sensor in the air pump. The positional PID controller is designed to compensate the error in air pressure as follows: $p_{d} (k) = K_{P}^{p} e_{p} (k) + K_{I}^{p} \sum_{i = 0} e_{p} (i) + K_{D}^{p} [e_{p} (k) - e_{p} (k - 1)],$ (16)where $K_{P}^{p}$ , $K_{I}^{p}$ , and $K_{D}^{p}$ denote the parameters for proportion, integration, and differentiation terms in air pressure controller. In summary, the error in grasping force and air pressure compensates mutually to overcome the difficulty in the pneumatic-driven grasping system, such as multivariable, nonlinear, and long time-delay.

Evaluation metrics

To evaluate the proposed method, some metrics were used, including mean square error ( $MSE$ ) for KF, transient performance, and steady-state performance for closed-loop controller, such as maximum overshoot $δ$ , fluctuation times $N$ , adjustment time $t_{s}$ , and steady-state error $e_{s s}$ . $MSE$ is calculated by the following equation: $MSE = \frac{1}{K} \sum_{k = 0}^{K} |\bar{F} (k) - \hat{F} (k)| .$ (17)

$δ$ is calculated through the following equation: $δ = \frac{\hat{F} (t_{p}) - F_{d}}{F_{d}} \times 100 %,$ (18)where $t_{p}$ is the peak time of the control system. $N$ is half the number of times for the $\hat{F} (k)$ to cross the set value $F_{d}$ . The definition of $t_{s}$ is when $t \geq t_{s}$ , $|\hat{F} (t) - \hat{F} (\infty)| \leq F_{d} \times 5 %$ . $e_{s s}$ could be calculated from the following equation: $e_{s s} = \lim_{t \to \infty} e_{f} (t) .$ (19)

Experimental Results and Analysis

In this section, we report and analyze the experimental results to evaluate the performance of the proposed grasping force control method for pneumatic gripper, in both a qualitative and a quantitative manner.

Results for KF

In the first place, the effect of KF was evaluated in both qualitative and quantitative manner. Texture analyzer was adopted to apply different forces on the sensing unit of the matrix force sensor, from 0.5 to 5.0 N, and the interval was 0.5 N. The measured force and the predicted force after KF are shown in Figure 2 for visualization, and the results for $MSE$ are listed in Table 2.

FIG. 2.

Results for KF with different forces of texture analyzer applied to the matrix force sensor. (A) 0.5 N. (B) 1.0 N. (C) 1.5 N. (D) 2.0 N. (E) 2.5 N. (F) 3.0 N. (G) 3.5 N. (H) 4.0 N. (I) 4.5 N. (J) 5.0 N.

Table 2.

Quantitative Results of the Kalman Filter with Different Forces of Texture Analyzer Applied to the Matrix Force Sensor

Force (N)	0.5	1.0	1.5	2.0	2.5	3.0	3.5	4.0	4.5	5.0
$MSE$	0.0756	0.0951	0.0849	0.1071	0.1006	0.1145	0.1468	0.1502	0.1670	0.2020

MSE, mean square error.

As seen from Figure 2, the predicted force signal after KF was smoother. It is obvious that the noise component in the force signal has been filtered. Meanwhile, the convergence rate of the predicted force signal was satisfactory when a step force signal was applied on the force sensor. The response speed could meet the requirements of real-time control. These results demonstrate that the noise covariance matrices $Q$ , $H$ , and $R$ are reasonable. In Table 2, the $MSE$ showed an increasing trend when the applied force increased. This demonstrates that the noise component increased when the applied force increased. The filtered force signal was processed for further matrix force data visualization and grasping force control, as given in the following subsection.

Results for data visualization of the matrix force sensor

To simulate and analyze the grasping condition of peaches, a standard rubber ball was used for experiment and comparison. The diameter was 70 mm and the weight was 223 g. Different input air pressures were applied to the pneumatic soft gripper, from 0 to 90 kPa. The interval was 30 kPa. During the experiment, the air pressure was controlled by the internal controller in Figure 1C. The visualized matrix force data are shown in Figure 3.

FIG. 3.

Results for data visualization of the matrix force sensor on standard rubber ball. The pressure of the air pump was set to different values. The average grasping force and the channel number of the force sensing units were annotated in the figure. The darker the color, the greater the grasping force. (A) The pressure was 0 kPa. (B) The pressure was 30 kPa. (C) The pressure was 60 kPa. (D) The pressure was 90 kPa.

As seen from the first row in Figure 3, when the input air pressure increased, the bending angle of the pneumatic gripper increased. Accordingly, the average grasping force for both left gripper and right gripper also increased, and the difference between left and right was no more than 0.2 N. On the contrary, when the input air pressure changed, the grasping force distribution condition among the 16 sensing units also changed. The region of the measured grasping force moved from the middle to the tip of the gripper. This demonstrates that the primary contact point between the grasping objects and the gripper changed when the bending angle of the gripper changed. Therefore, it is difficult to achieve the grasping force sensing and control for the force sensor with the single sensing unit. This is one of the reasons that matrix force sensor was studied in this research.

For further validation and comparison, peaches with different diameters and weights were used for experiments. The visualized matrix force data with different input air pressures (from 0 to 100 kPa) are shown in Figure 4. The interval of the air pressure was 20 kPa, and the air pressure was also controlled by the internal controller.

FIG. 4.

Results for data visualization of the matrix force sensor on peaches. The pressure of the air pump was set to different values, and peaches with different sizes and weights were taken for experiment. (A–F) The diameter was 71 mm and the weight was 184 g. (G–L) The diameter was 75 mm and the weight was 202 g. (M–R) The diameter was 81 mm and the weight was 216 g.

As seen in Figure 4, the overall trends for average grasping force were similar with the results for standard rubber ball, while the changing process was more irregular than a rubber ball. Similarly, the grasping force distribution changed from the middle of the gripper to the tip of the gripper, as the input air pressure increased. Furthermore, the unbalance between left gripper and right gripper was more obvious, the maximum deviation was 0.6 N. The reasons for the above phenomenon are that the firmness, shape, and surface smoothness are different for the agricultural products. Even for a single fruit, the above properties show up differently in the contact regions of the left and right grippers. This influenced the contact between the grasping objects and the gripper and force sensor. Based on the above summary, the results for the closed-loop control of the grasping force are reported in the next subsection.

Results for closed-loop control of the grasping force

Ablation studies were conducted based on the proposed double closed-loop grasping force control method with matrix force sensor and dead band. During the experiment, the matrix force sensor and the dead band were separately or simultaneously replaced with a single-channel force sensor and abandoned from the controller. The results are shown in Figure 5, and the quantitative evaluation metrics for both transient and steady-state performances are listed in Table 3. These metrics were calculated according to the definition in Eqs. (17)–(19).

FIG. 5.

Results for ablation study of the proposed closed-loop control method for the grasping force. The first column represents the results with constant desired grasping force. The second column represents the results with reduced desired grasping force. The third column represents the results with increased desired grasping force. (A–C) Represent the results without matrix force or dead band. (D–F) Represent the results with matrix force but without dead band. (G–I) Represent the results with dead band but without matrix force. (J–L) Represent the results with matrix force and dead band.

Table 3.

Ablation Study of the Proposed Closed-Loop Grasping Force Control Method for Pneumatic Gripper

Matrix force	Dead band	Desired grasping force (N)	Transient performance			Steady-state performance
Matrix force	Dead band	Desired grasping force (N)	$δ$ (%) $↓$	$N$ $↓$	$t_{s}$ (s) $↓$	$e_{s s}$ (N) $↓$
×	×	1.0	132.67	∞	∞	∞
×	×	2.0	153.74	∞	∞	∞
√	×	1.0	79.33	∞	∞	∞
√	×	2.0	82.91	∞	∞	∞
×	√	1.0	126.67	4	12.24	0.01
×	√	2.0	118.39	7	27.54	0.10
√	√	1.0	74.52	3	8.76	0.04
√	√	2.0	83.77	3	7.69	0.07

Here, $↓$ represents the performance is better when the value is smaller.

Bold represents the best performance.

Compared with force sensor with the single-sensing unit, matrix force sensor could obtain the distribution of the grasping force reliably and prevent the changes in measured grasping force due to the changes in contact point. Without multichannel sensing units, fluctuation may occur in the convergent system, as shown in Figure 5G. Dead band could prevent the control system from frequent fluctuation and action. The response curve demonstrated that both the matrix force sensor and the dead band are helpful in improving the performance of the control system.

As seen in Table 3, the multiple sensing units could improve the transient performance of the control system, such as maximum overshoot $δ$ (reduced from 132.87% to 80.13%), fluctuation times $N$ (reduced from 5.5 to 3 times), and adjustment time $t_{s}$ (reduced from 19.89 to 8.23 s). On the contrary, the addition of the dead band could improve the steady-state performance of the control system, the steady-state error $e_{s s}$ reduced from $\infty$ N to 0.05 N. When both multiple sensing units and dead band were added into the controller, the system could obtain the best performance. For transient performance, the maximum overshoot, fluctuation times, and adjustment time were 74.52%, 3, and 8.76 s, respectively. The maximum steady-state error was 0.07 N. Moreover, the response curves for standard rubber ball and peaches grasping force control are listed in Figure 6 for comparison.

FIG. 6.

Results for closed-loop control of the grasping force for both standard rubber ball and peaches with different sizes. The first column is the grasping photograph, the second, third, and fourth columns are the constant, decreased, and increased desired grasping forces, respectively. (A–D) Standard rubber ball. (E–H) Peaches, the diameter was 71 mm and the weight was 184 g. (I–L) Peaches, the diameter was 75 mm and the weight was 202 g. (M–P) Peaches, the diameter was 81 mm and the weight was 216 g.

As seen from Figure 6, the system could achieve robust grasping force control for both standard rubber ball and peaches with various diameters and weights. While the performance on peaches was inferior than the performance on standard rubber ball. This result demonstrated that the surface smoothness and consistency of peaches are different and worse than the standard rubber ball. The reliable contact between the grasping objects and the sensor and gripper may be influenced.

Furthermore, the generalization performance of the proposed method was verified with different types of fruits and fruits with different ripeness levels. The disturbance resistance ability was also verified. The performance was compared with a two-finger electric rigid gripper. During the movement process of the manipulator, the steady-state error was also compared and analyzed. The results are shown in Figure 7 for demonstration.

FIG. 7.

Generalization verification of the proposed method. (A–F) Results for different types of fragile fruit, including mango, avocado, orange, pear, apple, and muskmelon. (G–J) Results for different ripeness levels of the same peach, from day 1 to day 3 in the postharvest stage. (K–M) Results for closed-loop control of the grasping force with disturbance. The time when the disturbance was applied was annotated in yellow color in the figure. (N, O) Results for closed-loop control of the grasping force with other external disturbances, such as contacts with residual leaves and stems. (P–R) Grasping force control performance comparison between the pneumatic-driven system and the electric-driven system. (S) Grasping force control performance comparison between the moving robotic manipulator during pick-and-place manipulation and the static robotic manipulator during grasping.

In Figure 7A–F, another six types of fruits were tested, including mango, avocado, orange, pear, apple, and muskmelon. The shapes, sizes, smoothness, and firmness are different, and the weights vary from 137 to 369 g. These apparent differences in appearance and attributes did not invalidate the proposed grasping force control method. The results showed that our method performed well, the fluctuation times $N$ were no more than four times and the adjustment time $t_{s}$ was less than 8 s. In Figure 7G–J, a peach stored at room temperature (26°C) was tested daily in the postharvest stage. With the process of natural dehydration, the weight reduced from 182 to 162 g, and the firmness also decreased rapidly. The proposed method could adapt fruit in different ripeness levels. Moreover, as the firmness decreased, the fluctuation times $N$ and the maximum overshoot $δ$ also decreased. These demonstrated that the viscoelasticity is helpful to improve the transient performance of the system. Therefore, our method could protect the fruit with higher ripeness levels from being damaged by inappropriate grasping force. In Figure 7K–M, disturbances were added artificially through pulling the gripper or the grasping objects, after the control system enters steady state. The experimental results illustrated that the system could return to steady state after being disturbed by the external environment. The fluctuation times $N$ were no more than two times, and the adjustment time $t_{s}$ was less than 5 s. Compared with the steady state before disturbance, the steady-state error $e_{s s}$ did not change significantly. This demonstrated that the control system has the disturbance resistance ability. In Figure 7N and O, other external disturbances were also evaluated, such as the contact with residual leaves and stems on the peach. The results show that the proposed method could generalize on this extreme condition and achieve robust closed-loop grasping force control. The average steady-state error increased from 0.04 to 0.09 N. In Figure 7P–R, we evaluated the maximum overshoot $δ$ and the steady-state error $e_{s s}$ of the two-finger electric rigid gripper, and compared with our pneumatic system. A similar grasping force control method was deployed on the electric grasping system (see Fig. 7P), and the experiment was repeated 10 times. Although the maximum overshoot of the electric system was relatively small (26.21% on average), the pneumatic system performed better on steady-state error (0.04 N on average), while the average error was 0.12 N on an electric rigid gripper. The pneumatic-driven system could achieve grasping force control with higher accuracy. In Figure 7S, for further deployment and application, we evaluated the error in grasping force control during the movement and pick-and-place process of the robotic manipulator. The experiment was also repeated 10 times. The average steady-state error increased from 0.04 to 0.06 N, compared with the manipulator in static state.

Discussion

In this study, a grasping force control method suitable for fragile fruit and pneumatic-driven soft gripper was proposed. In this section, we give a detailed discussion about the strengths and limitations in our study and point out the future research directions.

Strengths

(1)

A method was proposed to solve the problem in grasping force control for pneumatic-driven soft gripper.

The electric rigid gripper has been widely adopted for industrial parts’ grasping and manipulation. However, for fragile and deformable objects such as agricultural products and food, the softness and compliance of a rigid gripper are insufficient. This may cause damage or early-stage bruise to the grasping objects.³ The pneumatic-driven soft gripper has attracted considerable attention for its better softness and compliance, low cost, and ease of fabrication. Nevertheless, the closed-loop control of the grasping force for pneumatic gripper is difficult to achieve. The pneumatic gripper is a complex multivariable, nonlinear, and long time-delay control system, and it is difficult to set up its mathematical model. Most of the related studies in recent years performed open-loop control. To solve this problem and achieve damage-less robotic fragile fruit grasping, we proposed a closed-loop grasping force control method applicable to pneumatic gripper based on KF and double PID controller with dead band. The proposed method has a better transient state and steady-state performance. The grasping force control system could resist the disturbance and return to steady state. (2)

The proposed solution could address the issue when contact point between gripper and grasping object changes.

According to the principle of the pneumatic gripper, the bending angle and the stroke of the fingertip change when the input air pressure of the air pump changes. Hence, when the grasping force is adjusted according to the air pressure, the contact point between the grasping objects and the gripper may make a difference. Most of the related studies in recent years adopted force sensor with single sensing unit to measure the grasping force of pneumatic gripper. The measured grasping force is not reliable when the contact point changes. Other studies idealized and simplified the contact condition as a single contact point without movement.^47,48 To solve this problem, we propose to adopt matrix force sensor with a higher resolution and multiple sensing units. The grasping force distribution condition and grasping mechanism for peaches were also analyzed. Compared with the force sensor with single sensing unit, our system could obtain a better transient performance. Besides, the unexpected fluctuation during steady state was also prevented.

Limitations

(1)

The proposed method would become invalid when the contact point moved out from the sensing area.

To a certain extent, the proposed method depends on the condition and accuracy of the force sensor. Compared with a single-unit force sensor, the matrix force sensor has a larger sensing area, while the sensing area is insufficient compared with human or animal skin. The proposed method would become invalid when the contact point moves out from the sensing region. (2)

The transient performance of the control system needs to be further improved.

Compared with the electric rigid gripper, the grasping force control algorithm needs to be improved. Specifically speaking, the maximum overshoot $δ$ needs to be further reduced to avoid potential damage to fragile fruit. The fluctuation times $N$ and adjustment time $t_{s}$ also need to be further reduced to improve the efficiency of the system. (3)

Uneven grasping force between two grippers is caused by different shapes and smoothness of the fruit.

As we can see from the experimental results, the grasping force distribution was uneven between left and right grippers, which may cause damage or early-stage bruise to the fruit. The maximum difference was 0.6 N. This result was caused by the different shapes and surface smoothness of the fruit.

Future research directions

In future study, a skin-like sensor is supposed to be developed to achieve grasping force sensing with a larger area and higher reliability. Multiple air pumps will be controlled independently to achieve uneven distribution of the grasping force. In addition, a real-time slip and damage detection method based on visual and tactile multimodalities will be integrated to adjust the desired grasping force in a self-adaptive pattern.

Conclusions

In this research, we aim to achieve damage-less robotic fragile fruit grasping in postharvest industry for automatic uploading and packaging to solve the problems in labor shortage and an aging global population. A robotic grasping system with matrix force sensor was developed. A closed-loop grasping force control method applicable to pneumatic-driven soft gripper was proposed in this study, based on KF and double PID controller with dead band. The external loop and internal loop were force controller and air pressure controller, respectively. Experiments were conducted to validate the effectiveness of noise filtering for KF in the first place. The matrix force data were also visualized for exploration of the grasping mechanism. To solve the problems in contact point changing and the long time-delay of the pneumatic control system, a control strategy based on matrix force sensor with dead band was proposed. The proposed method could achieve robust grasping force control of peaches with various sizes, weights, shapes, and surface smoothness. Ablation studies were conducted to demonstrate the effectiveness of the proposed control strategy. For transient performance of the control system, the maximum overshoot $δ$ , fluctuation times $N$ , and adjustment time $t_{s}$ were 74.52%, 3, and 8.76 s, respectively. The maximum steady-state error was 0.07 N. The disturbance resistance ability of the grasping force control system was also demonstrated. In future study, skin-like force sensor with a larger sensing area is supposed to solve the problem in uneven distribution of the grasping force. Also, real-time slip and damage detection method will be integrated to adjust the desired force in a self-adaptive mode. In summary, this study provided a feasible solution for damage-less robotic fragile fruit grasping, which is potential to be deployed in postharvest industry.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This research was supported and funded by the National Natural Science Foundation of China under Grant Number U20A2019.

Supplementary Material

References

Chakraborty

, Subeesh

, Potdar

, et al. AI‐enabled farm‐friendly automatic machine for washing, image‐based sorting, and weight grading of citrus fruits: Design optimization, performance evaluation, and ergonomic assessment. J Field Robot, 2023.

Wang

, Jin

, Wang

, et al. Apple stem/calyx real-time recognition using YOLO-v5 algorithm for fruit automatic loading system. Postharvest Biol Technol, 2022; 185:111808.

Wang

, Wu

, Sun

, et al. Design, integration, and evaluation of a robotic peach packaging system based on deep learning. Comput Electron Agric, 2023; 211:108013.

Pettersson

, Davis

, Gray

, et al. Design of a magnetorheological robot gripper for handling of delicate food products with varying shapes. J Food Eng, 2010; 98(3):332–338.

Wang

, Wu

, Liu

, et al. PlantStereo: A high quality stereo matching dataset for plant reconstruction. Agriculture, 2023; 13(2):330.

Romanov

, Korostynska

, Lekang

, et al. Towards human-robot collaboration in meat processing: Challenges and possibilities. J Food Eng, 2022; 331:111117.

Fang

, Sun

, Wu

, et al. Multimode grasping soft gripper achieved by layer jamming structure and tendon-driven mechanism. Soft Robot, 2022; 9(2):233–249.

Levine

, Pastor

, Krizhevsky

, et al. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res, 2018; 37(4–5):421–436.

, Yu

, Ding

, et al. Visual–tactile fusion for transparent object grasping in complex backgrounds. IEEE Trans Robot, 2023; 39(5):3838–3856.

10.

Morrison

, Corke

, Leitner

. Learning robust, real-time, reactive robotic grasping. Int J Robot Res, 2020; 39(2–3):183–201.

11.

Stella

, Della Santina

, Hughes

. How can LLMs transform the robotic design process? Nat Mach. Intell, 2023:1–4.

12.

Zhang

, Guo

, Wang

, et al. Automated dissection of intact single cell from tissue using robotic micromanipulation system. IEEE Robot Autom Lett, 2023; 8(8):4705–4712.

13.

Wang

, Xing

, Ying

, et al. CGFNet: 3D convolution guided and multi-scale volume fusion network for fast and robust stereo matching. Pattern Recognit Lett, 2023; 173:38–44.

14.

Liu

, Tian

, Wang

, et al. Key technologies of plug tray seedling transplanters in protected agriculture: A review. Agriculture, 2023; 13(8):1488.

15.

Wang

, Inoue

, Hashimoto

, et al. Measuring viscoelasticity and friction of tempuras for robotic handling. J Food Eng, 2021; 310:110707.

16.

Cao

, Fang

H-S

, Liu

, et al. Suctionnet-1billion: A large-scale benchmark for suction grasping. IEEE Robot Autom Lett, 2021; 6(4):8718–8725.

17.

Mahler

, Matl

, Satish

, et al. Learning ambidextrous robot grasping policies. Sci Robot, 2019; 4(26):eaau4984.

18.

Fang

H-S

, Wang

, Gou

, et al. Graspnet-1billion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020.

19.

Lenz

, Lee

, Saxena

. Deep learning for detecting robotic grasps. Int J Robot Res, 2015; 34(4–5):705–724.

20.

Calli

, Singh

, Bruce

, et al. Yale-CMU-Berkeley dataset for robotic manipulation research. Int J Robot Res, 2017; 36(3):261–268.

21.

Fang

H-S

, Wang

, Fang

, et al. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Trans Robot, 2023; 39(5):3929–3945.

22.

, Zhai

D-H

, Xia

. A novel robotic pushing and grasping method based on vision transformer and convolution. IEEE Trans Neural Netw Learn Syst, 2023.

23.

, Zhai

D-H

, Xia

, et al. SE-ResUNet: A novel robotic grasp detection method. IEEE Robot Autom Lett, 2022; 7(2):5238–5245.

24.

Zhang

, Chen

, Li

, et al. Close the optical sensing domain gap by physics-grounded active stereo sensor simulation. IEEE Trans Robot, 2023; 39(3):2429–2447.

25.

Zhang

, Zhang

, Zhao

, et al. Nondestructive dimension sorting by soft robotic grippers integrated with triboelectric sensor. ACS Nano, 2022; 16(2):3008–3016.

26.

, Kroemer

, Su

, et al. A review of tactile information: Perception and action through touch. IEEE Trans Robot, 2020; 36(6):1619–1634.

27.

Sun

, Kuchenbecker

, Martius

. A soft thumb-sized vision-based sensor with accurate all-round force perception. Nat Mach Intell, 2022; 4(2):135–145.

28.

Friedl

, Roa

. CLASH—A compliant sensorized hand for handling delicate objects. Front Robot AI, 2019; 6:138.

29.

Friedl

, Roa

. Experimental evaluation of tactile sensors for compliant robotic hands. Front Robot AI, 2021; 8:704416.

30.

Liu

, Zhao

, Geng

, et al. Soft humanoid hands with large grasping force enabled by flexible hybrid pneumatic actuators. Soft Robot, 2021; 8(2):175–185.

31.

Zhu

, Chai

, Yong

, et al. Bioinspired multimodal multipose hybrid fingers for wide-range force, compliant, and stable grasping. Soft Robot, 2023; 10(1):30–39.

32.

Zhang

, Zhou

, Yi

, et al. A flexible swallowing gripper for harvesting apples and its grasping force sensing model. Comput Electron Agric, 2023; 204:107489.

33.

, Scharff

, Long

, et al. Modelling of soft fiber-reinforced bending actuators through transfer learning from a machine learning algorithm trained from FEM data. Sensor Actuat A-Phys, 2024; 368:115095.

34.

Aly

, Low

, Long

, et al. Tactile sensing for tissue discrimination in robotic meat cutting: A feasibility study. J Food Eng, 2024; 363:111754.

35.

Huang

, He

, Choi

, et al. Picking dynamic analysis for robotic harvesting of Agaricus bisporus mushrooms. Comput Electron Agric, 2021; 185:106145.

36.

Zheng

, Xie

, Zhang

, et al. Dexterous robotic grasping of delicate fruits aided with a multi-sensory e-glove and manual grasping analysis for damage-free manipulation. Comput Electron Agric, 2021; 190:106472.

37.

Wang

, Li

, et al. Grasping behavior of the human hand during tomato picking. Comput Electron Agric, 2021; 180:105901.

38.

Fan

, Yan

, Wang

, et al. Three-finger grasp planning and experimental analysis of picking patterns for robotic apple harvesting. Comput Electron Agric, 2021; 188:106353.

39.

Huang

, Zheng

, Zhang

, et al. Deep learning with tactile sequences enables fruit recognition and force prediction for damage-free grasping. Comput Electron Agric, 2023; 211:107985.

40.

Zhang

, Zhou

, Yan

, et al. Hardness recognition of fruits and vegetables based on tactile array information of manipulator. Comput Electron Agric, 2021; 181:105959.

41.

, Liu

. A bionic adaptive end-effector with rope-driven fingers for pear fruit harvesting. Comput Electron Agric, 2023; 211:107952.

42.

Gao

, Zhang

, et al. Development and evaluation of a pneumatic finger-like end-effector for cherry tomato harvesting robot in greenhouse. Comput Electron Agric, 2022; 197:106879.

43.

Tawk

, Sariyildiz

, Alici

. Force control of a 3D printed soft gripper with built-in pneumatic touch sensing chambers. Soft Robot, 2022; 9(5):970–980.

44.

Fiedler

, Ruppel

, Jonetzko

, et al. A low-cost modular system of customizable, versatile, and flexible tactile sensor arrays. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021: IEEE.

45.

Chen

, Peng

, Huang

, et al. Force distribution of thumb-index finger power-grasp during stable fruit grasp control. Comput Electron Agric, 2022; 198:107058.

46.

Yang

, Liu

, et al. Compliant grasping control for a tactile self-sensing soft gripper. Soft Robot, 2024; 11(2):230–243.

47.

Abbasi

, Nekoui

, Zareinejad

, et al. Position and force control of a soft pneumatic actuator. Soft Robot, 2020; 7(5):550–563.

48.

Sonar

, Gerratt

, Lacour

, et al. Closed-loop haptic feedback control using a self-sensing soft pneumatic actuator skin. Soft Robot, 2020; 7(1):22–29.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

63.44 MB

0.00 MB