Adaptive fuzzy optimal control using direct heuristic dynamic programming for chaotic discrete-time system

Abstract

In this paper, we aim to solve the optimal tracking control problem for the Henon Mapping chaotic system using the direct heuristic dynamic programming (DHDP) setting with filtered tracking error. The fuzzy logic system is used to approximate the long-term utility function. Compared with the results for chaotic discrete-time system, the cost of the controller is reduced. The Lyapunov analysis approach is utilized to prove the stability of the chaotic system. It is shown that the tracking error, the adaptation law and the control input retain the property of uniformly ultimate boundedness. A simulation example is given to demonstrate the effectiveness of the proposed approach.

Keywords

Chaotic system fuzzy logic systems direct heuristic dynamic programming adaptive control Lyapunov analysis

1. Introduction

In the past decade, the control problem of the chaotic system has attracted much attention. Some traditional strategies have been designed for chaotic systems. In Wang and He (2008), to analyze the dynamical behavior of fractional order unified system, a projective synchronization approach of fractional order unified system was proposed based on the stability criterion of linear systems. A high precision fast projective synchronization method was proposed in Wang et al. (2009) through introducing an impact factor for chaotic (hyperchaotic) systems. An adaptive scheme based on Gaussian network was provided in Zhang et al. (1998) for regulation of uncertain chaotic systems. Subsequently, a lot of adaptive control methods were proposed one after the other in Hua et al. (2005), Chen et al. (2009), Chen and Chen (2009), Tang et al. (2013), Li (2012) and Li (2013a) for uncertain nonlinear chaotic systems. Specifically, a great amount of control approaches using the fuzzy logic systems and the neural networks for uncertain nonlinear systems had attracted much attention (Chen et al., 2013, 2014; Hu, 2009, Hu et al., 2008, Hu and Ma, 2008; Yang et al., 2008, 2009b; Li et al., 2010, 2011, 2012b, 2014a, 2014b, Li, 2012, 2013, 2014, Tong and Li, 2003, Tong et al., 2005, 2010, 2011).

The above results are obtained for the chaotic systems in continuous-time. In Yamamoto et al. (2001), a delayed feedback control was developed for chaotic discrete-time systems. Lu et al. (2001)'s study adaptive design using backstepping for a class of discrete-time chaotic systems with known or unknown parameters. However, the current methods do not consider the controller cost problem. It is significant work to find the optimal control performance.

Reinforcement learning has held great intuitive appeal and has attracted considerable attention in the past. But only recently has it made major advancements by implementing the temporal different (TD) learning method. The reinforcement learning based adaptive critic neural network (NN) approach has emerged as a promising tool to develop optimal NN controllers due to its potential to find approximate solutions to dynamic programming, where a strategic utility function (a long-term system performance measure) can be optimized. By contrast, dynamic programming (DP) provides truly optimal solutions to nonlinear stochastic dynamic systems. Heuristic dynamic programming (HPD) was proposed in the 1970s and the ideas were firmed up in the early 1990s under the names of adaptive critic designs. The original proposition for HDP was essentially the same as the formulation of reinforcement learning (RL) using TD methods. Specifically, this formulation falls exactly into the Bellman equation. HDP and the adaptive critics in general train a network to associate input states with action values. The direct heuristic dynamic programming (DHDP) method is an approximate DP, which was inspired by action-dependent HDP (ADHDP) and has been applied to large-scale complex realistic applications. Once again, the basic idea in adaptive critic design is to adapt the weights of the critic network to make the approximating utility function satisfy the modified Bellman equation.

In Yang and Jagannathan (2012), the adaptive reinforcement learning controller was proposed by using the neural networks for unknown nonlinear discrete-time multiple-input-multiple-output (MIMO) systems with the external disturbances. In the design, an action network that is designed to produce optimal signal and a critic network that evaluates the performance of the action network. In Lin (2003), Tang and Liu (2013), two adaptive reinforcement learning schemes were studied for robot manipulator with uncertainties. By designing the direct heuristic dynamic programming, a nonlinear tracking control setting with filtered tracking error was designed in Yang et al. (2009a). The stability analysis of the tracking system is given based on the Lyapunov stability. It is shown that the closed-loop tracking error and the approximating neural network weight estimates are uniformly ultimate boundedness.

The above methods are very good to solve optimal control problem. But the optimal control strategy is not obtained for the discrete-time chaotic systems. Thus, this paper tries to address an optimal control problem for Henon mapping chaos model with unknown parameters. The fuzzy logic systems are used to approximate the uncertainties of the systems. Based on direct heuristic dynamic programming Yang et al. (2009a), an optimal controller is designed compared with the method in Lu et al. (2001). The Lyapunov method is employed to analyze the stability of the closed-loop system. A simulation example is provided to verify the effectiveness of the proposed approach.

2. System description

The problem of controlling a class of discrete-time chaotic systems by using the DHDP approach will be studied. Consider the Henon Mapping chaos model

x_{1} (k + 1) = x_{2} (k) x_{2} (k + 1) = - x_{2}^{2} (k) + 0.3 x_{1} (k) + 1.4 + u (k)

(1)

where

x_{i} (k) \in R m, i = 1, 2

are the state vector, and

u (k) \in R m

stands for the control input of the system.

According to the chaos model (equation (1)), consider the chaotic system without the control input $u (k)$ , the chaotic phenomenon is shown in Figure 1.

Figure 1.

The chaotic phenomenon.

The tracking error $e (k)$ is defined as

e (k) = x_{2} (k) - x_{d} (k)

(2)

where

x_{d} (k)

is a known smooth bounded function with the upper bound

{\bar{x}}_{d} (k)

. According to equation (1), it is easy to get

e (k + 1) = x_{2} (k + 1) - x_{d} (k + 1) = - x_{2}^{2} (k) + 0.3 x_{1} (k) + 1.4 + u (k) - x_{d} (k + 1)

(3)

Given the desired trajectory $x_{d} (k) \in R m$ and its delayed values, the filtered tracking error $\bar{e} (k) \in R m$ is described as

\bar{e} (k) = e (k) + λ_{1} e_{1} (k)

(4)

where

e_{1} (k)

is the delayed errors of

e (k)

. i.e.,

e_{1} (k) = e (k - 1)

, and λ₁ is the constant matrice selected. It can be rewritten into matrix form as

\bar{e} (k) = [Λ I_{m \times m}] e (k)

(5)

where

e (k) = [e_{1}^{T}, e T] T

, Λ = λ₁I_m×m ∈ R^m×m and I_m×m ∈ R^{m × m} is an identity matrix.

Rewriting equation (4) at k + 1, i.e.

\bar{e} (k + 1) = e (k + 1) + λ_{1} e_{1} (k + 1)

(6)

and substituting equation (3) into equation (6), the discrete-time nonlinear system can be written in terms of the filtered tracking error as

\bar{e} (k + 1) = - x_{2}^{2} (k) + 0.3 x_{1} (k) + 1.4 - x_{d} (k + 1) + λ_{1} e (k) + u (k)

(7)

3. Fuzzy approximation

The basic configuration of the fuzzy logic systems consists of fuzzifier, fuzzy rule base, fuzzy inference engine and defuzzifier. The fuzzy inference engine employs fuzzy rules to perform a mapping from an input linguistic vector $\underline{z} = [z_{1}, \dots, z_{p}] T \in R p$ to an output linguistic variable $y (\underline{z}) \in R$ . The fuzzy rule base consists of a collection of the fuzzy IF-THEN rules. The lth fuzzy IF-THEN rule is described by

R l : If z_{1} is φ_{1}^{l} and z_{2} is φ_{2}^{l}, \dots, and z_{p} is φ_{p}^{l} then y is Φ l, l = 1, \dots, K

(8)

where

φ_{i}^{l}, i = 1, \dots, p

and Φ^l are fuzzy sets characterized by fuzzy membership functions

μ_{φ_{i}^{l}} (z_{i})

and

μ_{Φ l} (y)

, respectively, K is the number of rules in the fuzzy rule base.

The fuzzy logic systems with the singleton fuzzifier, product inference engine and center-average defuzzifier are defined as follows

y (\underline{z}) = \frac{\sum_{l = 1}^{K} \bar{ω} l (Π_{i = 1}^{p} μ_{φ_{i}^{l}} (z_{i}))}{\sum_{l = 1}^{K} Π_{i = 1}^{p} μ_{φ_{i}^{l}} (z_{i})}

(9)

where

\bar{ω} l

is the point at which

μ_{Φ l} (\bar{y} l) = 1

. In(9), if the membership function

μ_{φ_{i}^{l}} (z_{i})

is fixed and

\bar{ω} l

is viewed as adjustable parameters, then equation (9) can be rewritten as

y (\underline{z}) = θ T ς (\underline{z})

(10)

where

θ = [\bar{θ} 1, \dots, \bar{θ} K] T

is a vector of adjustable parameters and

ς (\underline{z}) = [ς 1 (\underline{z}), \dots, ς K (\underline{z})] T

is a fuzzy basis function vector where

ς l (\underline{z})

is defined as

ς l (\underline{z}) = \frac{Π_{i = 1}^{n} μ_{φ_{i}^{l}} (z_{i})}{\sum_{l = 1}^{K} Π_{i = 1}^{n} μ_{φ_{i}^{l}} (z_{i})}

(11)

From the definition of

ς (\underline{z})

, it is obvious to obtain

ς T (\underline{z}) ς (\underline{z}) \leq 1

(12)

It has been proven that the fuzzy logic systems can uniformly approximate any a nonlinear discrete function to an arbitrary accuracy on a compact set. We introduce the following lemma.

Lemma 1

For any a given function $f (\underline{z})$ on a compact set U∈R^p and an arbitrary ɛ > 0, there exists a fuzzy logic system in the form of equation (9) such that ${sup}_{\underline{z} \in U} | f (\underline{z}) - y (\underline{z}) | < ɛ$ .

Given a compact set $\underline{z} \in Ω_{\underline{z}}$ , based on Lemma 1, we know that the smooth function $f (\underline{z})$ can be expressed as

f (\underline{z}) = θ T ς (\underline{z}) + ɛ (\underline{z})

(13)

where θ* is an optimal fuzzy parameter vector, and the approximation error

ɛ (\underline{z})

satisfies

| ɛ (\underline{z}) | \leq \bar{ɛ}

with the constant

\bar{ɛ} > 0

In this paper, the control objective is to design an adaptive controller for the system (equation (1)) that all the signals in the closed-loop system remain uniformly ultimately bounded, the system state $x (k)$ follows a desired trajectory $x_{d} (k)$ , and the long-term cost function $Q (k)$ is minimized such that an optimal (or near optimal) control input can be generated.

4. Strategic utility function

Reinforcement signal $q (k)$ is defined based on the current filtered tracking error $\bar{e} (k)$ as

q (k) = [q_{1} (k), q_{2} (k), \dots, q_{m} (k)] \in R m

with

q_{i} (k) = {0, if ‖ {\bar{e}}_{i} (k) ‖ \leq c 1, if ‖ {\bar{e}}_{i} (k) ‖ > c, i = 1, 2, \dots, m

where constant c is a predefined threshold for the filtered tracking error, and

‖ \cdot ‖

is the Euclidean vector 2-norm. Signal

q_{i} (k)

is as simple as either a “0” or a “1” corresponding to “good” or “poor” tracking performance, respectively.

The goal of DHDP control is to optimize a long-term cost function $Q (k)$ given by

Q (k) = q (k + 1) + α q (k + 2) + α 2 q (k + 3) \dots

where α > 0 is a constant weighting factor.

We use the fuzzy logic system to estimate the long-term cost function

Q (k) = θ_{c}^{* T} φ_{c} (x (k)) + ɛ_{c}^{*} (x (k)) = θ_{c}^{* T} φ_{c} (k) + ɛ_{c}^{*} (k)

where

θ_{c}^{*}

is the weight matrix and

{\overset{\land}{θ}}_{c} (k)

is the estimation of

θ_{c}^{*}

, and the estimation error is defined as

{\tilde{θ}}_{c} (k) = {\overset{\land}{θ}}_{c} (k) - θ_{c}^{*}

. The activation function

φ_{c} (x (k))

is denoted as

φ_{c} (k)

for simplicity.

ɛ_{c}^{*} (x (k))

represents the approximation error which is denoted as

ɛ_{c}^{*} (k)

for simplicity. And the approximation error

ɛ_{c}^{*} (k)

is bounded by

{\bar{ɛ}}_{c}

, i.e.,

‖ ɛ_{c}^{*} (k) ‖ \leq {\bar{ɛ}}_{c}

By writing the Bellman equation into the prediction error

e_{c} (k) = α \overset{\land}{Q} (k) - [\overset{\land}{Q} (k - 1) - q (k)]

(14)

where

\overset{\land}{Q} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k)

is the estimation of

Q (k)

The objective function to be minimized based on the prediction error is defined as

E_{c} (k) = \frac{1}{2} e_{c}^{T} (k) e_{c} (k) = \frac{1}{2} e_{c}^{2} (k)

(15)

The weight update rule is given by

{\overset{\land}{θ}}_{c} (k + 1) = {\overset{\land}{θ}}_{c} (k) + Δ {\overset{\land}{θ}}_{c} (k) Δ {\overset{\land}{θ}}_{c} (k) = τ_{c} [- \frac{\partial E_{c} (k)}{\partial {\overset{\land}{θ}}_{c} (k)}] = τ_{c} [- \frac{\partial E_{c} (k)}{\partial e_{c} (k)} \frac{\partial e_{c} (k)}{\partial \overset{\land}{Q} (k)} \frac{\partial \overset{\land}{Q} (k)}{\partial {\overset{\land}{θ}}_{c} (k)}]

where τ_c > 0 is the learning rate.

By substituting equation (14) into the preceding equation, the weight-updating rule becomes

{\overset{\land}{θ}}_{c} (k + 1) = {\overset{\land}{θ}}_{c} (k) - τ_{c} α φ_{c} (k) \times (α {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) + q (k) - {\overset{\land}{θ}}_{c}^{T} (k - 1) φ_{c} (k - 1)) T

(16)

5. Controller design and stability analysis

Define control input $u (k)$ as

u (k) = x_{d} (k + 1) - (- x_{2}^{2} (k) + 0.3 x_{1} (k) + 1.4) + k_{v} \bar{e} (k) - λ_{1} e (k) - \overset{\land}{Q} (k)

(17)

where k_v is a diagonal gain matrix. The closed-loop error system then becomes

\bar{e} (k + 1) = k_{v} \bar{e} (k) - \overset{\land}{Q} (k)

(18)

Assuming that the filtered tracking error system is stable, given $0 < k_{v max} < 1$ , where $k_{v max}$ is the maximum eigenvalue of the constant gain matrix k_v.

Assumption 1

Bounded optimal target weight: With $θ_{c}^{*}$ as the optimal target weight, assume that it is bounded, so that $‖ θ_{c}^{*} ‖ \leq {\bar{θ}}_{c}$ , where ${\bar{θ}}_{c} \in R +$ represents the bound on the unknown target weight.

Remark 1

In contrast to a great amount of works on chaotic systems, few results are combined the DHDP techniques into the design process. This paper focuses on a tracking control structure using DHDP as an online learning control element. Specifically, we design a control input (17), which contains the filtered tracking error and the cost function. Once the long-term cost function is minimized, an optimal control input will be generated.

Now, the following theorem is explained to show the stability of the closed-loop system.

Theorem 1

Consider the discrete-time system given by equation (1). Let the control input is described as equation (17) and the adaptation law is shown by equation (16). Under the assumption 1, if the design parameters are chosen appropriately, the proposed algorithm can guarantee that all the signals are semiglobally uniformly ultimately bounded (SGUUB) and the tracking error $e (k)$ converges to a compact set described by ${lim}_{k \to \infty} ‖ x (k) - x_{d} (k) ‖ \leq δ$ where δ is a positive bounded constant.

Proof

Consider the following Lyapunov function

V (k) = V_{1} (k) + V_{2} (k) + V_{3} (k)

where

V_{1} (k) = \frac{1}{τ_{1}} \bar{e} (k) T \bar{e} (k), V_{2} (k) = \frac{γ_{c}}{τ_{c}} {\tilde{θ}}_{c}^{T} (k) {\tilde{θ}}_{c} (k), V_{3} (k) = \frac{1}{τ_{2}} ‖ ω_{c} (k - 1) ‖ 2 .

According to the filtered tracking error (equation (18)), the first difference of $V_{1} (k)$ is

Δ V_{1} (k) = \frac{1}{τ_{1}} (\bar{e} (k + 1) T \bar{e} (k + 1) - \bar{e} (k) T \bar{e} (k)) = \frac{1}{τ_{1}} (k_{v} \bar{e} (k) - \overset{\land}{Q} (k)) 2 - \frac{1}{τ_{1}} ‖ \bar{e} (k) ‖ 2

(19)

Invoke the Cauchy-Schwarz inequality described as

(a_{1} + \dots + a_{n}) 2 \leq n (a_{1}^{2} + \dots + a_{n}^{2})

(20)

Using the Cauchy-Schwarz inequality (equation (20)) and the equation $\overset{\land}{Q} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k)$ , (19) can be rewritten as

Δ V_{1} (k) \leq \frac{2}{τ_{1}} k_{v}^{2} ‖ \bar{e} (k) ‖ 2 + \frac{2}{τ_{1}} \overset{\land}{Q} 2 (k) - \frac{1}{τ_{1}} ‖ \bar{e} (k) ‖ 2 \leq \frac{4}{τ_{1}} (θ_{c}^{* T} φ_{c} (k)) 2 + \frac{4}{τ_{1}} ‖ ω_{c} (k) ‖ 2 + (\frac{2}{τ_{1}} k_{v}^{2} - \frac{1}{τ_{1}}) ‖ \bar{e} (k) ‖ 2

using

‖ φ_{c} (k) ‖ \leq 1

and

‖ θ_{c}^{*} ‖ \leq {\bar{θ}}_{c}

, the above inequation becomes

Δ V_{1} (k) \leq (\frac{2}{τ_{1}} k_{v max}^{2} - \frac{1}{τ_{1}}) ‖ \bar{e} (k) ‖ 2 + \frac{4}{τ_{1}} {\bar{θ}}_{c}^{2} + \frac{4}{τ_{1}} ‖ ω_{c} (k) ‖ 2

(21)

where

ω_{c} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) - θ_{c}^{* T} φ_{c} (k) = {\tilde{θ}}_{c}^{T} (k) φ_{c} (k)

is the approximation error of the cost function.

By subtracting $θ_{c}^{*}$ on both sides of equation (16), it obtains

{\tilde{θ}}_{c} (k + 1) = {\tilde{θ}}_{c} (k) - τ_{c} α φ_{c} (k) \times (α {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) + q (k) - {\overset{\land}{θ}}_{c}^{T} (k - 1) φ_{c} (k - 1)) T

(22)

The first difference of $V_{2} (k)$ is

Δ V_{2} (k) = \frac{γ_{c}}{τ_{c}} ({\tilde{θ}}_{c}^{T} (k + 1) {\tilde{θ}}_{c} (k + 1) - {\tilde{θ}}_{c}^{T} (k) {\tilde{θ}}_{c} (k))

(23)

Substituting equation (22) into the above equation, we have

Δ V_{2} (k) = γ_{c} [{\tilde{θ}}_{c} (k) - τ_{c} α φ_{c} (k) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) T] T \times [{\tilde{θ}}_{c} (k) - τ_{c} α φ_{c} (k) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) T] / τ_{c} - γ_{c} {\tilde{θ}}_{c}^{T} (k) {\tilde{θ}}_{c} (k) / τ_{c} = - γ_{c} (1 - τ_{c} α 2 ‖ φ_{c} (k) ‖ 2) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2 - 2 γ_{c} α ω_{c} (k) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) T + γ_{c} (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2

(24)

Based on the equation $\overset{\land}{Q} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k)$ and $ω_{c} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) - θ_{c}^{* T} φ_{c} (k)$ , (24) can be expressed as

Δ V_{2} (k) = - γ_{c} (1 - τ_{c} α 2 ‖ φ_{c} (k) ‖ 2) \times (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2 - γ_{c} α 2 ω_{c}^{2} (k) + γ_{c} (α θ_{c}^{* T} φ_{c} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2

(25)

Based on $‖ φ_{c} (k) ‖ \leq 1$ , (25) can be rewritten as

Δ V_{2} (k) \leq - γ_{c} (1 - τ_{c} α 2) \times (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2 + γ_{c} (α θ_{c}^{* T} φ_{c} (k) + q (k) - ω_{c} (k - 1) - θ_{c}^{* T} φ_{c} (k - 1)) 2 - γ_{c} α 2 ω_{c}^{2} (k)

(26)

Then, using the Cauchy-Schwarz inequality and the inequation $‖ θ_{c}^{*} ‖ \leq {\bar{θ}}_{c}$ , equation (26) can be expressed as

Δ V_{2} (k) \leq - γ_{c} (1 - τ_{c} α 2) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2 - γ_{c} α 2 ‖ ω_{c} (k) ‖ 2 + 2 γ_{c} ‖ ω_{c} (k - 1) ‖ 2 + 2 γ_{c} ({\bar{θ}}_{c} (1 + α) + 1) 2

(27)

The first difference of $V_{3} (k)$ is

Δ V_{3} (k) = \frac{1}{τ_{2}} (‖ ω_{c} (k) ‖ 2 - ‖ ω_{c} (k - 1) ‖ 2)

(28)

Based on the above analysis, let the first difference of the Lyapunov function be

Δ V (k) = Δ V_{1} (k) + Δ V_{2} (k) + Δ V_{3} (k)

Substituting equations (21), (27) and (28) into the above equation. By assumption, select

τ_{c} α 2 < 1 0 < k_{v max} < \sqrt{2} / 2 α > \sqrt{2} γ_{c} > 0

(29)

and

τ_{1} > 4 / (γ_{c} α 2 - 2 γ_{c}), τ_{2} = 1 / 2 γ_{c}

Finally, we get the first difference of $V (k)$ is

Δ V (k) \leq - γ_{c} (1 - τ_{c} α 2) (α \overset{\land}{Q} (k) + q (k) - \overset{\land}{Q} (k - 1)) 2 - (\frac{1}{τ_{1}} - \frac{2}{τ_{1}} k_{v max}^{2}) ‖ \bar{e} (k) ‖ 2 - (γ_{c} α 2 - \frac{1}{τ_{2}} - \frac{4}{τ_{1}}) ‖ ω_{c} (k) ‖ 2 + D

where

D = 2 γ_{c} ({\bar{θ}}_{c} (1 + α) + 1) 2 + \frac{4}{τ_{1}} ‖ θ_{c}^{*} ‖ 2

Given that conditions (equation (29)) hold and

‖ \bar{e} (k) ‖ > \sqrt{τ_{1} D / (1 - 2 k_{v max}^{2})}, ‖ ω_{c} (k) ‖ > \sqrt{D / (γ_{c} α 2 - \frac{1}{τ_{2}} - \frac{4}{τ_{1}})}

the first difference

Δ V (k) \leq 0

. According to the standard Lyapunov extension theorem, this demonstrates that the filtered tracking error

\bar{e} (k)

and the weight estimate

{\tilde{θ}}_{c} (k)

are SGUUB.

6. Simulation example

In this section, an example is provided to demonstrate the effectiveness of the tracking control and the filtered tracking error is proposed in this paper. In order to illustrate the feasibility of the theoretic results, the Henon Mapping chaos model will be considered. In addition to a fuzzy controller will be designed based on the DHDP.

The control objective is to design an adaptive fuzzy controller for the system (equation (1)) that all the signals in the closed-loop system remain SGUUB, the system state vector $x (k)$ tracks a desired trajectory $x_{d} (k)$ .

In this paper, the control system framework of the nonlinear tracking control with filtered tracking error has been implemented for DHDP design and analyzed using the Lyapunov stability approach, and a fuzzy logic system is applied to approximate the long-term cost function $Q (k)$ .

The Henon Mapping chaos system has been described in equation (1), and the simulation parameters are set as λ₁ = 0.25, τ_c = 0.01, α = 0.5, c = 0.1 and k_v = 0.5. The system initial states are chosen as $x 1 (0) = 0.1, x 2 (0) = 0.1$ . Based on the above design, the reinforcement signal, the filtered tracking error and the controller are shown as follows

{\overset{\land}{θ}}_{c} (k + 1) = {\overset{\land}{θ}}_{c} (k) - τ_{c} α φ_{c} (k) \times (α {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) + q (k) - {\overset{\land}{θ}}_{c}^{T} (k - 1) φ_{c} (k - 1)) T \overset{\land}{Q} (k) = {\overset{\land}{θ}}_{c}^{T} (k) φ_{c} (k) \bar{e} (k + 1) = k_{v} \bar{e} (k) - \overset{\land}{Q} (k) u (k) = x_{d} (k + 1) - (- x_{2}^{2} (k) + 0.3 x_{1} (k) + 1.4) + k_{v} \bar{e} (k) - λ_{1} e (k) - \overset{\land}{Q} (k)

Based on the chaotic system (equation (1)) and the fuzzy controller (equation (17)), the simulation results are shown in Figures 2–4. The tracking performance is described in Figure 2 and it can be seen that a good tracking trajectory is achieved. The adaptive law of θ_c is given in Figure 3. Last, Figure 4 shows the control input of the chaotic system and it can be observed from Figure 4 that the control input is bounded. It is easy to see that the proposed fuzzy controller make the closed-loop system stable.

Figure 2.

x₂(k) (solid line) and x_d(k) (dashed line).

Figure 3.

The adaptive law θc.

Figure 4.

The control input u .

7. Conclusion

This paper makes full use of the DHDP design in the optimal tracking control setting with filtered tracking error. By using the reinforcement learning, an adaptive control was proposed to solve the tracking problem for the Henon Mapping chaotic system. The long-term cost function which is approximated by a fuzzy logic system is minimized so that an optimal control input can be achieved. Results have shown that the tracking error and the adaptation law were uniformly bounded in the sense of Lyapunov. A simulation example has been applied to verify the effectiveness of the method.

Footnotes

Funding

This research was supported by the Natural Science Foundation of China under grant number 61104017.

References

Chen

(2009) Robust adaptive neural network synchronization controller design for a class of time delay uncertain chaotic systems. Chaos, Solitons & Fractals 41: 2716–2724.

Chen

Jiang

(2009) Sliding mode synchronization controller design with neural network for uncertain chaotic systems. Chaos, Solitons & Fractals 39: 1856–1863.

Chen CLP, Liu YJ and Wen GX (2013) Fuzzy neural network-based adaptive control for a class of uncertain nonlinear stochastic systems. IEEE Transactions on Cybemetics 44: 583–593.

Chen CLP, Wen GX, Liu YJ and Wang FY (2014) Adaptive consensus control for a class of nonlinear multi-agent time-delay systems using neural networks. IEEE Transactions On Neural Networks and Learning Systems. Epub ahead of print 6 February 2014. DOI: 10.1109/TNNLS.2014.2302477.

(2009) Robust adaptive sliding mode attitude maneuvering and vibration damping of three-axis-stabilized flexible spacecraft with actuator saturation limits. Nonlinear Dynamics 54: 301–321.

(2008) Adaptive variable structure controller for spacecraft vibration reduction. IEEE Transactions on Aerospace and Electronic Systems 44: 861–876.

Xie

(2008) Robust and adaptive variable structure output feedback control of uncertain systems with input nonlinearity. Automatica 44: 552–559.

Hua

Guan

Shi

(2005) Adaptive feedback control for a class of chaotic systems. Chaos, Solitons & Fractals 23: 757–765.

(2012) Adaptive output feedback control of uncertain nonlinear chaotic systems based on dynamic surface control technique. Nonlinear Dynamics 68: 235–243.

10.

Li DJ (2013a) Adaptive neural network control for unified chaotic systems with dead-zone input. Journal of Vibration and Control. Epub ahead of print 25 November 2013. DOI: 10.1177/1077546313513055.

11.

Li DJ (2013b) Adaptive neural network control for a class of continuous stirred tank reactor systems. Science China Information Sciences. Epub ahead of print 1 April 2014. DOI: 10.1007/s11432-013-4824-7.

12.

(2014) Neural network control for a class of continuous stirred tank reactor process with dead-zone input. Neurocomputing 131: 453–459.

13.

Hilton

Liu

(2013) Adaptive sliding mode control for nonlinear active suspension systems using T-S fuzzy model. IEEE Trans. Industrial Electronics 60: 3328–3338.

14.

Li H, Gao H, Shi P and Zhao X (2014a) Fault-tolerant control of markovian jump stochastic system with augmented sliding mode observer approach. Automatica, [in press].

15.

Jing

Karimi

(2014b) Output-feedback based H-infinity control for active suspension systems with control delay. IEEE Trans. Industrial Electronics 61: 436–446.

16.

Gao

Ding

(2011) Adaptive fuzzy control for synchronization of nonlinear teleoperators with stochastic time-varying communication delays. IEEE Trans, Fuzzy System 19: 745–757.

17.

Kang

(2010) Adaptive robust coordinated control of multiple mobile manipulators interacting with rigid environments. Automatica 46: 2028–2034.

18.

Yang

Ding

Bogdan

(2012) Robust adaptive motion control for remotely operated vehicles with velocity constraints. International Journal of Control, Automation, and System 10: 421–429.

19.

Lin

(2003) A reinforcement learning adaptive fuzzy controller for robots. Fuzzy Sets and Systems 137: 339–352.

20.

Wei

Wang

(2001) Backstepping control of discrete time chaotic systems with application to the Henon systems. IEEE Trans. Circ. Syst. I 48: 1359–1363.

21.

Tang L and Liu YJ (2013) Adaptive neural network control of robot manipulator using reinforcement learning. Journal of Vibration and Control. Epub ahead of print 21 June 2013. DOI: 10.1177/1077546313482171.

22.

Tang L, Liu L and Liu YJ (2013) Adaptive control design for Arneodo chaotic system with state constraint. Journal of Vibration and Control. Epub ahead if print 18 September 2013. DOI: 1077546313504980.

23.

Tong

(2003) Fuzzy adaptive sliding-mode control for MIMO nonlinear systems. IEEE Transactions on Fuzzy Systems 11: 354–360.

24.

Tong

Chen

Wang

(2005) Fuzzy adaptive output feedback control for MIMO nonlinear systems. Fuzzy Sets and Systems 156: 285–299.

25.

Tong

Liu

(2011) Observer-based adaptive fuzzy backstepping control for a class of stochastic nonlinear strict-feedback systems. IEEE Transactions on Systems, Man and Cybernetics, Part B 41: 1693–1704.

26.

Tong

Liu

(2010) Fuzzy-adaptive decentralized output-feedback control for large-scale nonlinear systems with dynamical uncertainties. IEEE Transactions on Fuzzy Systems 18: 845–861.

27.

Wang

(2008) Projective synchronization of fractional order chaotic system based on linear separation. Physics Letters A 372: 435–441.

28.

Wang

Nian

Guo

(2009) High precision fast projective synchronization in chaotic (hyperchaotic) systems. Physics Letters A 373: 1754–1761.

29.

Yamamoto

Hino

Ushio

(2001) Dynamic delayed feedback controllers for chaotic discrete time systems. IEEE Trans. Circ. Syst. I 48: 785–789.

30.

Yang

Xiang

(2008) Output feedback NN control for two classes of discrete-time systems with unknown control directions in a unified approach. IEEE Transactions on Neural Networks 19: 1873–1886.

31.

Yang

Tsakalis

Rodriguez

(2009a) Direct heuristic dynamic programming for nonlinear tracking control with filtered tracking error. IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics 39: 1617–1622.

32.

Yang

Lee

(2009b) Output feedback adaptive control of a class of nonlinear discrete-time systems with unknown control directions. Automatica 45: 270–276.

33.

Yang

Jagannathan

(2012) Reinforcement learning controller design for affine nonlinear discrete-time systems using online approximators. IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics 42: 377–390.

34.

Zhang

Qin

Chen

(1998) Adaptive control of chaotic systems with uncertainties. Int. J. Bifurcation Chaos 8: 2041–2046.