Robust online model predictive control for image-based visual servoing in polar coordinates

Abstract

In this paper, a novel robust online model predictive control (RMPC) method for image-based visual servoing (IBVS) in polar coordinates is proposed. First, the Jacobian matrix is transformed into a weighted combination of vertex matrices of convex polytopic by tensor product (TP) model transformation method. Then, a new IBVS control design condition for 6-DOF manipulator submitting to robot physical limitations and visibility constraints is obtained in polar coordinates by using RMPC technique. The optimal value of the control signal can be solved online when carrying out the convex optimization problem. The proposed control strategy can effectively improve the trajectory in the case that involves translation and rotation with fast response while averting the pseudo-inverse of the image Jacobian matrix. Conclusively, the effectiveness of the proposed scheme is validated by simulations and experiments on a 6-DOF manipulator.

Keywords

Classic image-based visual servoing (IBVS)polar-based IBVS tensor product (TP) model model predictive controller (MPC)

Introduction

Visual servoing (VS) has attracted more and more attention of researchers from various fields of robots, such as manipulators (Cai et al., 2013, 2016; Li and Zhao, 2017), mobile robots (Ke et al., 2017; Wang et al., 2009; Zhang et al., 2018), quadrotors (Islam et al., 2015; Thomas et al., 2016) and underwater vehicles (Gao et al., 2015, 2017). VS control may be divided into two types in the light of the feedback signals of the closed-loop system: (1) position-based visual servoing (PBVS) (Park et al., 2012; Thuilot et al., 2002); (2) image-based visual servoing (IBVS) (Tahri et al., 2013; Xie et al., 2009). The advantage of PBVS (Chaumette and Hutchinson, 2006) is that both visual error signal and input signal of the system are spatial pose. Path planning and controller design are relatively simple, and singular values of the robot can be avoided. However, since the image signal is independent of the control loop formed by Cartesian space, it is uncertain whether the reference object is in the camera’s field of view in the course of the whole visual servoing task. Compared with PBVS, the image data is directly employed to control manipulators in IBVS, so that pose estimation is not required whilst it is robust to camera and hand–eye calibration errors (Allibert et al., 2010). Therefore, IBVS is mostly studied by scholars nowadays.

Although vision-based control of IBVS is a robust method for the error of target depth, it does not produce the optimal Cartesian path for large rotation around the optical axis. In order to remedy the problem mentioned above, polar-based IBVS method is proposed (Chaumette and Hutchinson, 2007). Similar to the classic IBVS method, in Iwatsuki and Okiyama (2005) and Corke et al. (2009), a proportional controller is proposed by calculating the pseudo-inverse of Jacobian matrix to obtain the control errors in polar coordinates. It is observed that an appropriate velocity screw is spanned for rotational motion around the optical axis, solving the issue on camera degradation in IBVS. To take advantages of IBVS and polar-based IBVS, respectively, a class of switching controllers is proposed whose switching signals are excited in different ways. For example, the switching signal is offered by a prediction of the camera displacement in Allibert and Courtial (2012). In Ye et al. (2015), it is given on the basis of task decomposition using information from homography.

There are few literatures on polar-based IBVS method for the following reasons. On the one hand, it only guarantees local stability and may have a defect of Jacobian matrix singularity, which may lead visual servoing task to failure. Some partitioned techniques have been recommended to settle these problems on the singularity of Jacobian matrix (Hajiloo et al., 2016; Hao et al., 2007; Wang et al., 2015; Yüksel, 2017; Zhang et al., 2017). In Hao et al. (2007), a controller based on Takagi–Sugeno fuzzy neural network approach is presented for IBVS system, which can avoid image Jacobian matrix singularity. However, visual constraints in visual servoing tasks need further consideration. An approach on effectively combining extreme learning machine (ELM) and fuzzy logic (FL) is presented and applied to the visual servoing system successfully in Yüksel (2017). The pseudo-inverse of the Jacobian matrix is estimated by applying trained ELMs. It is worth mentioning that FL unit is activated to obtain reverse linear velocities to drag the features away from the boundary of camera view. In Zhang et al. (2017), the visual servoing problem is translated to a constrained optimization problem involving joint angle and velocity limits of the manipulator via a recurrent neural network, but real-time effect is not ideal. To improve real-time performance in visual servoing controller, a new IBVS control design condition for 6-DOF robotic visual servoing system submitting to robot physical limitations and visibility constraints is proposed in Wang et al. (2015) and Hajiloo et al. (2016). The optimal value of the control signal could be solved online when carrying out the convex optimization problem.

On the other hand, another significant reason for limiting the use of polar-based IBVS method is that it is generally not as effective as the classical IBVS method when the camera performs other motions (except rotating around the optical axis). In order to conquer aforementioned shortcomings as far as possible, a robust online model predictive control method for image-based visual servoing in polar coordinates is presented. First, VS system is expressed as a discrete time domain linear parameter varying (LPV) state space model. By tensor product (TP) model transformation, a weighted combination of vertex matrices of convex polytopic for image Jacobian matrix is obtained. Then, VS task is realized by working out convex optimization problem of linear matrix inequalities (LMIs) online rolling submitting to robot physical limitations and visibility constraints. The proposed control strategy can effectively improve the trajectory in which case that involves translation and rotation. In the case of large rotation motion with translation motion, especially in the x, y directions, the advantages of the polar-based IBVS method can be fully developed. In addition, the problem of Jacobian matrix singularity in the classic polar-based IBVS method is overcome. Finally, the effectiveness of the proposed scheme is validated by simulations and experiments on a 6-DOF manipulator with eye-in-hand configuration.

The paper is organized as follows. In the second section, classic IBVS and polar-based IBVS methods are described, respectively. In the third section, LPV model for polar-based IBVS and TP model transformation are presented. Then, an RMPC algorithm for polar-based IBVS is introduced in the fourth section. In the fifth section, several simulation and experiment results are presented to verify the effectiveness of the proposed control scheme. Finally, conclusions are provided in the sixth section.

Polar-based IBVS and its state space model

Classic IBVS

By using the central projection imaging model, a 3-D point $P_{i} = (X_{i}, Y_{i}, Z)$ in the world frame is projected onto the 2-D image plane according to the following formula.

p_{ci} = (x_{i}, y_{i}) = (X_{i} / Z, Y_{i} / Z)

(1)

where $p_{ci}$ is the normalized image plane coordinates. The relationship between pixel coordinates $s_{ci}$ and plane coordinates $p_{ci}$ of the normalized image is expressed as:

s_{ci} = (\begin{matrix} u_{i} \\ v_{i} \end{matrix}) = (\begin{matrix} \frac{f}{ρ_{u}} & 0 \\ 0 & \frac{f}{ρ_{v}} \end{matrix}) (\begin{matrix} \frac{X_{i}}{Z} \\ \frac{Y_{i}}{Z} \end{matrix}) + (\begin{matrix} u_{0} \\ v_{0} \end{matrix})

(2)

where focal length is denoted as $f$ , $(u_{0}, v_{0})^{T}$ represents the principal point, $ρ_{u}$ and $ρ_{v}$ are the width and height of each pixel respectively. The relationship between ${\overset{\cdot}{s}}_{ci}$ and $V$ is represented by (Janabi-Sharifi et al., 2011):

{\overset{\cdot}{s}}_{ci} = L_{sc} V

(3)

where

L_{sc} = (\begin{matrix} - \frac{f}{ρ_{u} Z} & 0 & \frac{\bar{u}}{Z} & \frac{ρ_{u} \bar{u} \bar{v}}{f} & - \frac{f^{2} + ρ_{u}^{2} {\bar{u}}^{2}}{ρ_{u} f} & \bar{v} \\ 0 & - \frac{f}{ρ_{v} Z} & \frac{\bar{v}}{Z} & \frac{f^{2} + ρ_{v}^{2} {\bar{v}}^{2}}{ρ_{v} f} & - \frac{ρ_{v} \bar{u} \bar{v}}{f} & - \bar{u} \end{matrix})

(4)

is the image Jacobian matrix or interaction matrix, $V = [υ_{x}, υ_{y}, υ_{z}, ω_{x}, ω_{y}, ω_{z}]^{T} \in R^{6 \times 1}$ represents the control input, which consists of linear and angular velocity.

Polar-based IBVS

When an image feature point $p_{ci} = [x_{i}, y_{i}]^{T}$ , $i = 1, \dots, n$ are redefined in polar coordinates, a feature point $s_{pi} = [r_{i}, ϕ_{i}]^{T}$ can be obtained, where $r_{i}$ is the distance from the principal point to the current point (Chaumette and Hutchinson, 2007):

r_{i} = \sqrt{{\bar{u}}^{2} + {\bar{v}}^{2}}

(5)

$\bar{u}$ and $\bar{v}$ are image coordinates relative to the principal point. The angle between the u-axis and the line connecting the principal point and the current image point is:

ϕ_{i} = \arctan \frac{\bar{v}}{\bar{u}}

(6)

By substituting $x_{i}$ by $r_{i} \cos ϕ_{i}$ and $y_{i}$ by $r_{i} \sin ϕ_{i}$ , a new Jacobian matrix in polar coordinates can be obtained. Accordingly, the relationship between ${\overset{\cdot}{s}}_{pi}$ and $V$ is given by:

{\overset{\cdot}{s}}_{pi} = L_{sp} V

(7)

where

Ł_{sp} = (\begin{matrix} - \frac{f}{Z} \cos ϕ_{i} & - \frac{f}{Z} \sin ϕ_{i} & \frac{r_{i}}{Z} & \frac{f^{2} + r_{i}^{2}}{f} \sin ϕ_{i} & - \frac{f^{2} + r_{i}^{2}}{f} \cos ϕ_{i} & 0 \\ \frac{f}{r_{i} Z} \sin ϕ_{i} & - \frac{f}{r_{i} Z} \cos ϕ_{i} & 0 & \frac{f}{r_{i}} \cos ϕ_{i} & \frac{f}{r_{i}} \sin ϕ_{i} & - 1 \end{matrix})

(8)

In this paper, we choose four feature points in order to control the 6-DOF robot manipulator (Hashimoto and Noritsugu, 1998). That is, $n = 4$ . Usually, the current image feature vector is represented as $s_{p} = [s_{p 1}; s_{p 2}; s_{p 3}; s_{p 4}] \in R^{8 \times 1}$ , and $s_{p}^{*} \in R^{8 \times 1}$ represents the expected image feature vector. The image features error vector $e = [e_{1}; e_{2}; e_{3}; e_{4}] \in R^{8 \times 1}$ is defined as:

e_{i} = s_{pi} - s_{pi}^{*} = (\begin{matrix} r_{i} - r_{i}^{*} \\ ϕ_{i} ⊖ ϕ_{i}^{*} \end{matrix}) i = 1, \dots, 4

(9)

where ⊖ is the modulo $2 π$ subtraction for the angular component. Note, $| r | >> | ϕ |$ , hence normalization is necessary. A proportional controller for polar-based IBVS is designed in general for sake of decreasing the image errors exponentially:

\overset{\cdot}{e} = - λ e, λ > 0

(10)

Combining (7) and (10), the resulting control law can be expressed as:

V = - λ {\hat{L}}_{sp}^{+} e

(11)

where ${\hat{L}}_{sp}^{+}$ is the pseudo-inverse of $L_{sp}$ . As can be indicated in the Jacobian matrix (8), the radius $r_{i}$ is unaltered for rotation about the z-axis. Furthermore, the polar angle $ϕ_{i}$ is constant for translation $T_{z}$ along the optical axis. All the above characteristics of Jacobian matrix explain why polar-based IBVS method produces better camera motion for large rotation, but not for translation. For purpose of improving the performance of classic polar-based IBVS, a robust online model predictive control strategy is developed.

LPV model for polar-based IBVS and TP model transformation

An LPV model for polar-based IBVS and tensor product (TP) model transformation for convex polyhedron decomposition of LPV model are presented in this section. In order to adopt robust online model predictive control (RMPC) method for robotic systems, transforming the VS system into polyhedron form is indispensable.

Time varying parameters in the Jacobian matrix

All variables in (8) except the camera focal length $f$ are change with the progress of the visual servoing task. $r (t)$ , $ϕ (t)$ , $Z (t)$ and $L_{sp} (t)$ are time varying variables of $r$ , $ϕ$ , $Z$ and $L_{sp}$ in visual servoing respectively. Denote the range of $r (t)$ as $[r_{\min}, r_{\max}]$ and the range of $ϕ (t)$ as $[ϕ_{\min}, ϕ_{\max}]$ . It is well known that depth parameter $Z$ is hard to be acquired online. However, the depth value in this paper is considered as a time varying variable which can be measured by the initial and expected pose of the camera. $Z_{\min}$ and $Z_{\max}$ represent the minimum and maximum depth values, respectively.

So the Jacobian matrix can be regarded as a closed hypercube $Ω = [r_{\min}, r_{\max}] \times [ϕ_{\min}, ϕ_{\max}] \times [Z_{\min}, Z_{\max}] \subset R^{3}$ with respect to the variables $r$ , $ϕ$ , $Z$ in the time-varying range. For convenience, the three variables are denoted by a parameter vector $Γ (t) = [Γ_{1} (t), Γ_{2} (t), Γ_{3} (t)] = [r (t), ϕ (t), Z (t)]$ , then the Jacobian matrix can be expressed as $L_{sp} (Γ (t))$ .

Tensor product model transformation for Jacobian matrix

TP model transformation is applied to the Jacobian matrix in this subsection, which can minimize the number of vertices of the decomposed convex polyhedron model, and each weighting function is a single variable function of the parameter vector $Γ (t)$ .

The first step of obtaining TP model is to define an N-dimensional hyper rectangular equidistant grid-by-grid net over the closed hypercube $Ω$ (Baranyi et al., 2013) to discretize the LPV model. We obtain:

g_{n, m_{n}} = A_{n} + \frac{B_{n} - A_{n}}{M_{n} - 1} (m_{n} - 1), n = 1, \dots, N

(12)

where $N$ is the number of variable parameters in the Jacobian matrix, which is equal to 3 in this paper. $M_{n}$ represents the total number of grids corresponding to n dimensions. $m_{n}$ stands for the position of the grid at the current moment. $A_{n}$ and $B_{n}$ represent the minimum and maximum values of parameter variables selected on each dimension, respectively. Also, $A_{n} ⩽ g_{n, m_{n}} ⩽ B_{n}, m_{n} = 1, \dots, M_{n}$ (Baranyi et al., 2013).

Then, the Jacobian matrix $L_{sp}$ is sampled over grid points as follows:

£_{m 1, m 2, m 3}^{d} = L_{s p} (Γ_{m 1, m 2, m 3})

(13)

where $Γ_{m 1, m 2, m 3} = g_{1, m 1}, g_{2, m 2}, g_{3, m 3}$ . Superscript ‘d’ means ‘discrete’. $M_{1} \times M_{2} \times M_{3}$ different Jacobian matrices $£_{m 1, m 2, m 3}^{d}$ can be obtained from closed hypercube $Ω$ via formula (13).

The corresponding Jacobian matric becomes:

£_{m 1, m 2, m 3}^{s} = \sum_{m_{1} = 1}^{M_{1}} \sum_{m_{2} = 1}^{M_{2}} \sum_{m_{3} = 1}^{M_{3}} \prod_{n = 1}^{3} ω_{n, m_{n}} (Γ_{n} (t)) £_{m 1, m 2, m 3}^{d}

(14)

where $ω_{n, m_{n}} (Γ_{n} (t))$ is a weighting function of sampling values of interval $[A_{n}, B_{n}]$ in n-dimension. $£_{m 1, m 2, m 3}^{d} \in R^{2 \times 6}$ is the vertex matrix of the hypercube $Ω$ . Based on (14), the $(N + 2)$ dimensional tensor $£_{m 1, m 2, m 3}^{s} \in R^{M_{1} \times M_{2} \times M_{3} \times 2 \times 6}$ can be represented by the combination of vertex matrices and weighting functions.

In order for the TP model to be convex, the weighting function for all $Γ (t) \in Ω$ must satisfy the following conditions:

\forall n, m, Γ_{n} (t) ω_{n, m_{n}} (Γ_{n} (t)) ⩾ 0

(15)

\forall n, Γ_{n} (t) \sum_{m_{n} = 1}^{M_{n}} ω_{n, m_{n}} (Γ_{n} (t)) = 1

(16)

Next, high-order singular value decomposition (HOSVD) for each dimension of $£_{m 1, m 2, m 3}^{s}$ is necessary to be performed. In order to reduce computational load, we need to discard appropriate quantity of the singular values by trading off the accuracy and the complexity of HOSVD calculation. The error $φ$ between the accurate tensor $£_{m 1, m 2, m 3}^{s}$ and reduced one ${\hat{£}}_{m 1, m 2, m 3}^{s}$ can be expressed as (Baranyi et al., 2003):

φ = ∥ £_{m 1, m 2, m 3}^{s} - {\hat{£}}_{m 1, m 2, m 3}^{s} ∥_{2} = \sum_{i = 1}^{d} o_{i}

(17)

where $o_{i}$ is the discarded singular values and $d$ represents the number of discarded singular values.

At present, TP model which is obtained by the above operation lays the foundation for the robust online model predictive controller design for polar-based IBVS.

Robust online model predictive controller design

The robust online model predictive control (RMPC) controller for polar-based IBVS is proposed in this section. The control strategy can be applied to polar-based IBVS system to plan the optimal path for each step of motion trajectory, which improves trajectory in the case that involves translation and rotation whilst avoiding the influence of the Jacobian matrix singularity. Finally, the stability of the controller is proved by Lyapunov stability theorem.

Discrete-time LPV model for polar-based IBVS

A discrete polyhedron LPV model for polar-based IBVS according to (7) is considered as:

s_{pi} (ε + 1) = I s_{pi} (ε) + L_{e} (Γ_{i} (ε)) V (ε)

(18)

where $s_{pi} = [r_{i}, ϕ_{i}]$ is the each image point in polar coordinates. Also, $L_{e} (Γ_{i} (ε)) = T L_{sp} (Γ_{i} (ε))$ represents the Jacobian matrix and $T$ is the sampling time. In (18), $L_{e} (Γ_{i} (ε))$ is the function of the time-varying parameter vector $Γ_{i} (ε)$ and I is a $2 \times 2$ constant matrix.

Referring to continuous-time systems, the error of feature points in discrete-time systems is defined as $e_{i} (ε) = s_{pi} (ε) - s_{pi}^{*}$ , $i = 1, 2, 3, 4$ . $ε$ is the sampling time and $s_{pi}^{*}$ is the vector of expected feature points. The discrete time domain LPV state space model of visual servoing system (19) can be obtained by using the system state variable $e_{i} (ε)$ .

x_{i} (ε + 1) = x_{i} (ε) + L_{e} (Γ_{i} (ε)) V (ε)

e_{i} (ε) = x_{i} (ε)

(19)

where $L_{e} (Γ_{i} (ε)) = T L_{sp} (Γ_{i} (ε)), i = 1, 2, 3, 4$ .

RMPC design for polar-based IBVS

For the visual servoing system (19), the parameter vector $Γ_{i} (ε)$ and state variable $x_{i} (ε)$ can be measured in real time. The current input matrix $L_{e} (Γ_{i} (ε))$ can be obtained at any sampling time $ε$ by the analytical form of the Jacobian matrix. It is known that uncertain future input matrices $L_{e} (Γ_{i} (ε + j))$ , $j ⩾ 1$ vary in the convex polyhedron $Ω$ . Robot physical limitations and visibility constraints need to be considered as well. In order to effectively combine model uncertainty with system constraints, an RMPC method is designed for the input control signal $V (ε)$ so that an upper bound of infinite horizon objective function at sampling time $ε$ can be minimized (Lu and Arkun, 2000).

\begin{matrix} min_{V_{i 0}^{\infty} (ε)} max_{Le (Γ_{i} (ε + j)) \in Ω} J_{i 0}^{\infty} (ε) \\ J_{i 0}^{\infty} (ε) = x_{i} (ε | ε)^{T} Q x_{i} (ε | ε) + V (ε | ε)^{T} RV (ε | ε) \\ + \sum_{j = 1}^{\infty} x_{i} (ε + j | ε)^{T} Q x_{i} (ε + j | ε) + V_{i} (ε + j | ε)^{T} R V_{i} (ε + j | ε) \end{matrix}

(20)

where $Q > 0$ and $R > 0$ are weighting matrices. For convenience, also make $J_{1}^{\infty} = \sum_{j = 1}^{\infty} x_{i} (ε + j | ε)^{T} Q x_{i} (ε + j | ε) + V_{i} (ε + j | ε)^{T} R V_{i} (ε + j | ε)$ . Considering the current time control vector and future time control vectors separately, the control vector can be defined as $V_{i 0}^{\infty} = [V (ε | ε), V_{i 1}^{\infty}]$ , where $V (ε | ε) = V (ε)$ is the current time control signal. The future time control sequence $V_{i 1}^{\infty} = V_{i} (ε + j | ε), j ⩾ 1$ is calculated by the following state feedback law.

V_{i} (ε + j | ε) = F_{i} (ε) x_{i} (ε + j | ε), j ⩾ 1

(21)

It can be seen that since $V (ε | ε)$ is certain, the upper bound on the worst value of objective function $J_{i 0}^{\infty} (ε)$ in formula (20) is transformed into the upper bound on the worst value of objective function $J_{1}^{\infty} (ε)$ by an approach given in Kothare et al. (1996). Choose a quadratic function $Φ (χ) = χ^{T} P χ, P > 0$ , where $χ$ represents a substitute variable. For any $L_{e} (Γ_{i} (ε + j)) \in Ω, j ⩾ 1$ , assume that $Φ (χ)$ satisfies the following inequalities:

\begin{matrix} Φ (x_{i} (ε + j | ε)) - Φ (x_{i} (ε + j + 1 | ε)) \\ ⩾ x_{i} (ε + j | ε)^{T} Q x_{i} (ε + j | ε) + V_{i} (ε + j | ε)^{T} R V_{i} (ε + j | ε) \end{matrix}

(22)

Let $x_{i} (\infty | ε) = 0$ , $Φ (x_{i} (\infty | ε)) = 0$ is obtained further. Summing (22) from $j = 1$ to $j = \infty$ , $J_{1}^{\infty} (h)$ follows that

max_{Le (Γ_{i} (ε + j)) \in Ω} J_{1}^{\infty} (ε) ⩽ x_{i} (ε + 1 | ε)^{T} P (ε) x_{i} (ε + 1 | ε)

(23)

Therefore, the optimal problem (20) is converted to the following minimization question:

min_{V (ε | ε), γ, P} \sum γ

(24)

subject to:

\begin{matrix} x_{i} (ε | ε)^{T} Q x_{i} (ε | ε) + V (ε | ε)^{T} RV (ε | ε) + x_{i} (ε + 1 | ε)^{T} P (ε) x_{i} \\ (ε + 1 | ε) ⩽ γ \end{matrix}

(25)

The vertex set of $L_{e} (Γ_{i} (ε + j))$ can be defined by closed hypercube $Ω$ . By applying Schur complement lemma to inequality (22), the formula (30) can be obtained. Similarly, (29) can be obtained by inequality (25).

Considering robot physical limitations and visibility constraints to the system (19), the system input and output constraints are expressed as:

\begin{matrix} | V_{l} (ε | ε) | ⩽ V_{l, \max} \\ ∥ V_{i} (ε + j | ε) ∥_{2} ⩽ V_{\max} \end{matrix}

(26)

\begin{matrix} ∥ e_{i} (ε + j | ε) ∥_{2} ⩽ e_{\max} \\ ∥ e_{ri} (ε + j | ε) ∥_{2} ⩽ e_{rmax} \\ ∥ e_{ϕ i} (ε + j | ε) ∥_{2} ⩽ e_{ϕ \max} \end{matrix}

(27)

where $V_{\max}$ and $e_{\max}$ are the upper bound on the input vector $V_{i} (ε + j | ε)$ and the output vector $e_{i} (ε + j | ε)$ , respectively. $e_{ri} (ε + j | ε)$ and $e_{ϕ i} (ε + j | ε)$ are the components of the output vector $e_{i} (ε + j | ε)$ . The dimension of current input signal $V (ε | ε)$ is expressed as $l = 1, 2, \dots, 6$ . By applying Schur complement lemma to inequality (26)–(27), (31)–(36) can be obtained.

Theorem 1. For the visual servoing system (19) with the input and output constraints (26) and (27), by minimizing the upper bound of the infinite horizon objective function via solving the convex optimization problem of linear matrix inequalities (LMI), the errors of all image feature points will converge to zero: where $l = 1, \dots, 6$ , $i = 1, 2, 3, 4$ , and the symbol * induces a symmetric structure. $L_{er} = L_{e} (Γ_{i} (ε + j)), j ⩾ 1$ represents the Jacobian matrix of all vertices, $r = 1, \dots, D$ , and $D$ represents the number of vertices.

min_{V (ε | ε), γ, Q_{ω}, Y_{ω}} \sum γ

(28)

subject to

(\begin{matrix} - 1 & * & * & * \\ - (x_{i} (ε | ε) + L_{e} (Γ_{i} (ε)) V (ε | ε)) & - Q_{ω} & * & * \\ - Q^{0.5} x_{i} (ε | ε) & 0 & - γ I & * \\ - R^{0.5} V (ε | ε) & 0 & 0 & - γ I \end{matrix}) ⩽ 0

(29)

(\begin{matrix} - Q_{ω} & * & * & * \\ - (Q_{ω} + L_{er} Y_{ω}) & - Q_{ω} & * & * \\ - Q^{0.5} Q_{ω} & 0 & - γ I & * \\ - R^{0.5} Y_{ω} & 0 & 0 & - γ I \end{matrix}) ⩽ 0

(30)

| V_{l} (ε | ε) | ⩽ V_{l, \max}

(31)

(\begin{matrix} Y_{ω} & V_{\max}^{2} \\ Q_{ω} & * \end{matrix}) ⩽ 0

(32)

∥ x_{i} (ε | ε) + L_{e} (Γ_{i} (ε)) V (ε | ε) ∥_{2} ⩽ e_{\max}

(33)

∥ e_{ri} (ε + j | ε) ∥_{2} ⩽ e_{rmax}

(34)

∥ e_{ϕ i} (ε + j | ε) ∥_{2} ⩽ e_{ϕ \max}

(35)

(\begin{matrix} {(Q_{ω} + L_{er} Y_{ω})}^{T} & Q_{ω} \\ e_{\max}^{2} & * \end{matrix}) ⩽ 0

(36)

Proof. Choose a Lyapunov function

ϒ (ε) = \sum_{i = 1}^{4} ϒ_{i} (ε)

(37)

where

\begin{matrix} ϒ_{i} (ε) = x_{i} (ε | ε)^{T} Q x_{i} (ε | ε) + V (ε | ε)^{T} RV (ε | ε) + x_{i} (ε + 1 | ε)^{T} \\ P (ε) x_{i} (ε + 1 | ε) \end{matrix}

Obviously $ϒ (ε) ⩾ 0$ . Assuming that linear matrix inequalities (29)–(36) are feasible with respect to (28) at time $ε$ , the optimal solutions of $V (ε | ε)$ and $P (ε)$ is available. From (22), when $j = 1$ , the following inequality can be acquired:

\begin{matrix} x_{i} (ε + 1 | ε)^{T} Q x_{i} (ε + 1 | ε) + V_{i} (ε + 1 | ε)^{T} R V_{i} (ε + 1 | ε) \\ + x_{i} (ε + 2 | ε)^{T} P (ε) x_{i} (ε + 2 | ε) ⩽ x_{i} (ε + 1 | ε)^{T} P (ε) x_{i} (ε + 1 | ε) \end{matrix}

(38)

Here, $V_{i} (ε + 1 | ε)$ and $P (ε)$ are only feasible solutions at sampling time $ε + 1$ . However, at sampling time, $V_{i} (ε + 1 | ε + 1)$ and $P (ε + 1)$ are optimal solutions of (28)–(36). Therefore, the following relationships are obtained:

\begin{matrix} x_{i} (ε + 1 | ε + 1)^{T} Q x_{i} (ε + 1 | ε + 1) \\ + V (ε + 1 | ε + 1)^{T} RV (ε + 1 | ε + 1) \\ + x_{i} (ε + 2 | ε + 1)^{T} P (ε + 1) x_{i} (ε + 2 | ε + 1) \\ ⩽ x_{i} (ε + 1 | ε)^{T} Q x_{i} (ε + 1 | ε) + V_{i} (ε + 1 | ε)^{T} R V_{i} (ε + 1 | ε) \\ + x_{i} (ε + 2 | ε)^{T} P (ε) x_{i} (ε + 2 | ε) \end{matrix}

(39)

Comparing (39) with (38), we have $x_{i} (ε | ε) \neq 0$ and $V (ε | ε) \neq 0$ :

\begin{matrix} ϒ_{i} (ε + 1) = x_{i} (ε + 1 | ε + 1)^{T} Q x_{i} (ε + 1 | ε + 1) \\ + V (ε + 1 | ε + 1)^{T} RV (ε + 1 | ε + 1) \\ + x_{i} (ε + 2 | ε + 1)^{T} P (ε + 1) x_{i} (ε + 2 | ε + 1) \\ ⩽ x_{i} (ε + 1 | ε)^{T} P (ε) x_{i} (ε + 1 | ε) \\ < x_{i} (ε + 1 | ε)^{T} P (ε) x_{i} (ε + 1 | ε) + x_{i} (ε | ε)^{T} Q x_{i} (ε | ε) \\ + V (ε | ε)^{T} RV (ε | ε) = ϒ_{i} (ε) \end{matrix}

(40)

That is, $ϒ_{i} (ε + 1) < ϒ_{i} (ε)$ , $i = 1, 2, 3, 4$ . Thus,

ϒ (ε + 1) - ϒ (ε) = \sum_{i = 1}^{4} ϒ_{i} (ε + 1) - \sum_{i = 1}^{4} ϒ_{i} (ε) < 0

(41)

which meets the Lyapunov stability theorem. It implies the RMPC method can guarantee that each image point finally reaches the corresponding desired position.

Thus, the proof is complete.

Remark 1. It should be noted that the current control signal is only applied to the system (19) to drive the movement of the robot manipulator. Control sequences of the future moment which does not actually act on the system is merely used to guarantee the convergence of the future system state, to achieve global asymptotic stability of the system.

Remark 2. It is known that the objective function can be composed of the current time part and the future time part. Owing to the known current control signal $V (ε | ε)$ , the optimization problem is transformed into minimizing the upper bound on the part of objective function $J_{1}^{\infty}$ . Simultaneously, the current constraint (31) and constraints at future moment are considered separately, which is conducive to the feasibility of the input.

Remark 3. Compared with (Hajiloo et al., 2016), a new TP model based on the point characteristics of polar coordinates is established. In the process of model establishment, the characteristics of image features in polar coordinates need to be taken into account, and additional normalization processing is required. In combination with the RMPC method, a new visibility constraint strategy for polar coordinates is introduced because inherent characteristics of polar coordinates is reflected in the large difference between distance feature and angle feature errors. By implementing the scheme of constrains two kinds of errors respectively, the motion of image points out of camera field of view is more effectively avoided, and the motion trajectory of images can achieve the expected effect, which is also an additional consideration compared with the RMPC method for IBVS.

Simulation and experiment results

On behalf of verifying the effectiveness of the proposed controller, some simulations and experiments on the basis of a 6-DOF robot manipulator are carried out in this section.

The four points selected come into being a square with side length of 2 cm in Cartesian space, whose coordinates with respect to the camera base frame are (-0.01, -0.01, 1.17), (-0.01, 0.01, 1.17), (0.01, 0.01, 1.17), (0.01, -0.01, 1.17) m. Since the simulation represents an ideal situation, It is noted that there is almost no deviation between the camera and the end-effector frame. The relevant parameters of this camera which is mounted on the robot end-effector are shown in Table 1. For the 6-DOF Puma560 robot, the maximum linear and angular velocities are limited to 0.5 m/s and 0.5 rad/s, respectively.

Table 1.

Camera calibration parameters.

Parameters	Description	Values
$npix$	Camera resolution (pixel)	$374 \times 240$
$f$	Focal length (m)	0.004
$rho$	Normalized focal length on u,v axis (m)	$9.09 \times 10^{- 6}$
$u_{0}$	Principal coordinates of u axis (pixel)	187
$v_{0}$	Principal coordinates of v axis (pixel)	120
$ε$	Sampling time(s)	0.04

TP model transformation

The LPV model for polar-based IBVS is discretized by using equidistant grid lines in each dimension of hypercube $Ω$ . Then, a $100 \times 100 \times 100 \times 2 \times 6$ tensor of VS system is obtained by TP transformation. The range of closed hypercube is expressed as $Ω = [10^{- 5}, 0.6] \times [- π, π] \times [0.5, 2]$ . Nonzero singular values are preserved after applying HOSVD on each 3-D of the system tensor. Then, the Jacobian matrix can be represented as ` convex combination form of $4 \times 3 \times 2 = 24$ linear time-invariant vertex matrices.

\begin{matrix} L_{sp} (Γ (ε)) = \sum_{κ_{1} = 1}^{4} \sum_{κ_{2} = 1}^{3} \sum_{κ_{3} = 1}^{2} ω_{1, κ_{1}} (r (ε)) ω_{2, κ_{2}} (ϕ (ε)) ω_{3, κ_{3}} (Z (ε)) \\ L_{κ_{1}, κ_{2}, κ_{3}} \end{matrix}

(42)

where $ε$ is sampling time, $ω_{1, κ_{1}}$ , $κ_{1} = 1, 2, 3, 4$ , $ω_{2, κ_{2}}$ , $κ_{2} = 1, 2, 3$ , $ω_{3, κ_{3}}$ , $κ_{3} = 1, 2$ are weighting functions depicted in Figure 1.

Simulation results and analysis

Choose the weighting matrix $Q = I_{2 \times 2}$ and $R = I_{6 \times 6}$ . Four vertices of rectangular object are selected as visual feature points. In order to ensure that object points are always within the boundary of the phase plane in polar coordinates, the visual constraint is described as (43) and the velocity vector $V (ε | ε) \in R^{6 \times 1}$ is calculated by solving LMIs (29)–(36) online.

[\begin{matrix} r_{\min} = 0 \\ ϕ_{\min} = - π \end{matrix}] ⩽ s_{p} (ε) ⩽ [\begin{matrix} r_{\max} = 0.6 \\ ϕ_{\max} = π \end{matrix}]

(43)

The simulation results are compared with those of the classic polar-based IBVS method in Chaumette and Hutchinson (2007) and the RMPC scheme for IBVS in Hajiloo et al. (2016). (To facilitate comparison with IBVS, the simulation results are described in Cartesian coordinates.) Since the depth of classic polar-based IBVS method is difficult to be measured online, the expected depth is assumed as the actual depth value (Keshmiri et al., 2014). Subfigures of Figures 2 and 5 show the movement of the reference object from points labelled ‘green triangle’ to points labeled ‘red rectangle’.

In the first test, a visual servoing task involving large rotation and translation from the initial features to the expected features is performed. The initial and desired location of the feature points can be obtained from Table 2.

Table 2.

Initial (i) and desired (d) location of feature points in image plane (pixel).

		Point1	Point2	Point3	Point4
		( $u_{1} v_{1}$ )	( $u_{2} v_{2}$ )	( $u_{3} v_{3}$ )	( $u_{4} v_{4}$ )
Test1	i	41.23 41.11	61.39 88.81	109.1 68.64	88.92 20.95
	d	127.8 87.85	127.8 152.2	192.2 152.2	192.2 87.85
Test2	i	148.7 118.8	159.8 125.9	166.9 114.9	155.8 107.8
	d	127.8 87.85	127.8 152.2	192.2 152.2	192.2 87.85
Test3	i	350.0 104.2	399.6 103.3	398.6 52.99	348.7 53.58
	d	341.4 90.57	395.2 88.62	392.7 33.87	339.0 35.68
Test4	i	349.5 103.5	399.5 102.5	398.1 52.09	348.5 53.07
	d	365.2 124.2	422.3 120.5	418.4 62.50	361.3 66.06

The results for Test 1 are displayed in Figures 2 –4, which feature trajectories of three methods during the VS task are given in Figure 2, respectively. It can be indicated that RMPC method for polar-based IBVS has better image 2D trajectories than other two methods at the same run time. In contrast to the RMPC method for polar-based IBVS, the trajectory of the RMPC method for IBVS is discontinuous near the desired location in Figure 2(c). Figure 3(b) shows that the convergence time is quite long and the change of speed is more uneven, although the classic polar-based IBVS method can make the visual servoing system converge in the end with sufficient time. The feature errors for three systems are given in Figure 3. The camera velocity curves of three methods during the VS task are given in Figure 4. As can be seen from the velocity curve, the velocity in Figure 4(b) is almost unchanged at 0. The response time of the proposed method is obviously faster than that of the classic polar-based IBVS method, which shows the advantage of fast response.

For sake of further demonstrating that the proposed method is resultful, another similar VS task from the initial features to the desired features is performed, in which the camera centre rotates 1 radian. Table 2 indicates the initial and desired location of the features. The results of this test is shown in Figures 5 –7.

As shown in Figure 5, the proposed polar-based IBVS controller not only has better feature trajectories than the other two, but also the convergence time is shorter. The feature errors for three systems are given in Figure 6. The camera velocity curves of three methods during the VS task are given in Figure 7. It can be clearly seen that the system response speed of the method proposed in this paper is faster than those of other two methods. Under the RMPC method for IBVS, it should be added that there is an oscillation phenomenon in the feature errors and camera velocity curves during the later period of motion, respectively.

Experiment results and analysis

In order to further prove the method proposed in this paper, some experiments are carried out on VP-6242M Denso robot, where a Logitech web camera C310 is mounted on the end-effector. The relevant parameters of this camera are shown in Table 3. The robot communicates with its controller with a frequency of 1 kHz and the camera capturing rate is 40 frames/second. Then, the experimental results comparison between the RMPC method for polar-based IBVS and classic polar-based IBVS method is provided.

Table 3.

Camera calibration parameters.

Parameters	Description	Values
$f_{x}$	Normalized focal length on u axis	817.88
$f_{y}$	Normalized focal length on v axis	815.34
$u_{0}$	Principal coordinates of u axis (pixel)	320
$v_{0}$	Principal coordinates of v axis (pixel)	240

In the Test 3, a visual servoing task involving translation and rotation on the z axis from the initial features labelled ‘green circle’ to the expected features labelled ‘red circle’ is performed. The initial and desired location of the feature points are obtained from Table 2.

The results for Test 3 are displayed in Figures 8 –10, which feature trajectories of two methods during the VS task shown in Figure 8, respectively. It can be indicated that the RMPC method for polar-based IBVS has smoother image 2D trajectories than the classic polar-based IBVS method. Figure 9 shows that there is longer convergence time in the classic polar-based IBVS method and feature error curves have more fluctuations.

In the Test 4, the servoing task increases the translation motion of x and y axes compared with the Test 3. Table 2 indicates the initial and desired location of the features. The results of this test are shown in Figures 11 –13. The proposed method is better than the classic polar-based IBVS method, which is mainly reflected in that the motion trajectory is close to a straight line and the velocity oscillation frequency is weakened. The feature errors of two methods are given in Figure 12. The camera vecolity curves of two methods during the VS task are depicted in Figure 13.

In general, aforementioned simulation results fully verify that the RMPC method for polar-based IBVS proposed in this section can effectively accomplish given visual servoing tasks on the premise of dealing with robot physical limitations and visibility constraints. the proposed method is demonstrated that it is robust to constrains with satisfactory performance and improves the effect of translation motion. Although the experimental results are not significantly improved compared with the RMPC method for IBVS, it does solve the limitations of the classic polar-based IBVS method to some extent.

Conclusion

In this paper, a novel robust online model predictive control for IBVS in polar coordinates has been proposed. First, a convex hypercube decomposition method based on TP model transformation is applied to the LPV state space model of visual servoing system. Then, a new IBVS control design condition for 6-DOF manipulator submitting to robot physical limitations and visibility constraints is obtained in polar coordinates by using RMPC technique. The control signal of the robot manipulator is solved online according to the convex optimization problem of LMIs. Also, the error of servoing task is demonstrated to be asymptotically stable via using Lyapunov stability conditions. The proposed method has the superiority over other two performing in better control effect which is embodied in smooth trajectories with fast response and high accuracy. Finally, the simulation and experiment results of a 6-DOF manipulator verify the validity of the proposed method. Future research will concentrate on addressing more complex visual servoing tasks such as long distance motions.

Footnotes

Declaration of conflicting interest

The authors declare that there is no conflict of interests with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Natural Science Foundation of China under Grant 61873056, Grant 61473068, Grant 61621004 and Grant 61420106016, the Fundamental Research Funds for the Central Universities in China under Grant N170405004, N182608004 and the Research Fund of State Key Laboratory of Synthetical Automation for Process Industries in China under Grant 2013ZCX01.

ORCID iD

Jiuxiang Dong

References

Allibert

Courtial

(2012) Switching controller for efficient IBVS. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp.1695–1701.

Allibert

Courtial

Chaumette

(2010) Predictive control for constrained image-based visual servoing. IEEE Transactions on Robotics 26(5): 933–939.

Baranyi

Tikk

Yam

, et al. (2003) From differential equations to PDC controller design via numerical transformation. Computers in Industry 51(3): 281–297.

Baranyi

Yam

Várlaki

(2013) Tensor Product Model Transformation in Polytopic Model-based Control. Boca Raton: CRC Press.

Cai

Dean-León

Mendoza

, et al. (2013) Uncalibrated 3d stereo image-based dynamic visual servoing for robot manipulators. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp.63–70.

Cai

Somani

Knoll

(2016) Orthogonal image features for visual servoing of a 6-DOF manipulator with uncalibrated stereo cameras. IEEE Transactions on Robotics 32(2): 452–461.

Chaumette

Hutchinson

(2006) Visual servo control. I. Basic approaches. IEEE Robotics & Automation Magazine 13(4): 82–90.

Chaumette

Hutchinson

(2007) Visual servo control. II. Advanced approaches [Tutorial]. IEEE Robotics & Automation Magazine 14(1): 109–118.

Corke

Spindler

Chaumette

(2009) Combining Cartesian and polar coordinates in IBVS. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp.5962–5967.

10.

Gao

Proctor

, et al. (2017) Sliding mode adaptive neural network control for hybrid visual servoing of underwater vehicles. Ocean Engineering 142: 666–675.

11.

Gao

Proctor

Bradley

(2015) Adaptive neural network visual servo control for dynamic positioning of underwater vehicles. Neurocomputing 167: 604–613.

12.

Hajiloo

Keshmiri

Xie

, et al. (2016) Robust online model predictive control for a constrained image-based visual servoing. IEEE Transactions on Industrial Electronics 63(4): 2242–2250.

13.

Hao

Sun

Fujii

(2007) Image based visual servoing using takagi-sugeno fuzzy neural network controller. In: 2007 IEEE 22nd International Symposium on Intelligent Control. IEEE, pp.53–58.

14.

Hashimoto

Noritsugu

(1998) Performance and sensitivity in visual servoing. In: Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146), vol. 3. IEEE, pp.2321–2326.

15.

Islam

Liu

Saddik

(2015) Robust control of four-rotor unmanned aerial vehicle with disturbance uncertainty. IEEE Transactions on Industrial Electronics 62(3): 1563–1571.

16.

Iwatsuki

Okiyama

(2005) A new formulation of visual servoing based on cylindrical coordinate system. IEEE Transactions on Robotics 21(2): 266–273.

17.

Janabi-Sharifi

Deng

Wilson

(2011) Comparison of basic visual servoing methods. IEEE/ASME Transactions on Mechatronics 16(5): 967–983.

18.

Xiao

, et al. (2017) Visual servoing of constrained mobile robots based on model predictive control. IEEE Transactions on Systems, Man, and Cybernetics: Systems 47(7): 1428–1438.

19.

Keshmiri

Xie

Mohebbi

(2014) Augmented image-based visual servoing of a manipulator using acceleration command. IEEE Transactions on Industrial Electronics 61(10): 5444–5452.

20.

Kothare

Balakrishnan

Morari

(1996) Robust constrained model predictive control using linear matrix inequalities. Automatica 32(10): 1361–1379.

21.

Zhao

(2017) Global finite-time adaptive control for uncalibrated robot manipulator based on visual servoing. ISA transactions 68: 402–411.

22.

Arkun

(2000) Quasi-min-max MPC algorithms for LPV systems. Automatica 36(4): 527–540.

23.

Park

Kwon

(2012) Novel position-based visual servoing approach to robust global stability under field-of-view constraint. IEEE Transactions on Industrial Electronics 59(12): 4735–4752.

24.

Tahri

Araujo

Chaumette

, et al. (2013) Robust image-based visual servoing using invariant visual information. Robotics and Autonomous Systems 61(12): 1588–1600.

25.

Thomas

Loianno

Daniilidis

, et al. (2016) Visual servoing of quadrotors for perching by hanging from cylindrical objects. IEEE Robotics and Automation Letters 1(1): 57–64.

26.

Thuilot

Martinet

Cordesses

, et al. (2002) Position based visual servoing: keeping the object in the field of vision. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 2. IEEE, pp.1624–1629.

27.

Wang

Xie

Liu

, et al. (2015) Quasi-min-max model predictive control for image-based visual servoing with tensor product model transformation. Asian Journal of Control 17(2): 402–416.

28.

Wang

Lang

de Silva

(2009) A robust mobile robot manipulation system using a switch-based visual-servo controller. In: 2009 International Conference on Industrial Mechatronics and Automation. IEEE, pp.364–367.

29.

Xie

, et al. (2009) Switching control of image-based visual servoing with laser pointer in robotic manufacturing systems. IEEE Transactions on Industrial Electronics 56(2): 520–529.

30.

Wan

, et al. (2015) Novel two-stage hybrid IBVS controller combining Cartesian and polar based methods. In: 2015 IEEE International Conference on Mechatronics and Automation (ICMA). IEEE, pp.397–402.

31.

Yüksel

(2017) Intelligent visual servoing with extreme learning machine and fuzzy logic. Expert Systems with Applications 72: 344–356.

32.

Zhang

Chen

, et al. (2018) Unified visual servoing tracking and regulation of wheeled mobile robots with an uncalibrated camera. IEEE/ASME Transactions on Mechatronics 23(4): 1728–1739.

33.

Zhang

Liao

, et al. (2017) A recurrent neural network approach for visual servoing of manipulators. In: 2017 IEEE International Conference on Information and Automation (ICIA). IEEE, pp.614–619.