Reinforcement learning-based active fault diagnosis with tracking performance guaranteed

Abstract

Due to the consideration of control performance and the uncertainty of the dynamic characteristics of nonlinear systems, designing the auxiliary signal for active fault diagnosis presents significant challenges. This paper presents a novel data-driven approach for auxiliary signal design in the active fault diagnosis of nonlinear systems while ensuring guaranteed control performance. Specifically, we introduce a double actor-critic network to generate tracking and diagnostic signals, respectively. Subsequently, a two-objective optimization method based on deep reinforcement learning is proposed to address the tradeoff between tracking performance and fault diagnosis. Finally, the effectiveness of this method is verified through a cart-pole system with stochastic noise.

Keywords

Active fault diagnosis deep reinforcement learning actor-critic proximal policy optimization fault tolerant control

Introduction

In recent years, fault diagnosis has garnered significant attention and found widespread application in various domains such as cloud computing systems, power electronics, and chemical processes (Lee et al., 2022; Li et al., 2020; Liu et al., 2024; Taqvi et al., 2021; Zhao et al., 2024, 2025). Fault diagnosis can be categorized into passive and active diagnosis. Early research primarily focused on passive fault diagnosis (PFD), wherein the fault diagnosis is implemented by analyzing the system’s passive measured states/output information. Due to its passive nature, valuable diagnostic information may be overlooked as system uncertainties, and some faults may be masked by feedback controllers, thereby diminishing the accuracy of diagnosis.

To address this limitation, active fault diagnosis (AFD) designs an auxiliary signal that induces sufficient fault information within a given timeframe, while complying with system constraints. For diagnostic algorithms, both AFD and PFD are not very different and thoroughly surveyed (Blanke et al., 2006; Chiang et al., 2006; Venkatasubramanian et al., 2003a; 2003b; 2003c). Thus, the auxiliary signal design is a key research for AFD.

Generally speaking, there are two main research orientations of auxiliary signal design, which are the probabilistic method and the set-based method (Heirung and Mesbah, 2019). The main idea of the probabilistic method is to design an auxiliary signal that can maximize the distance of each probabilistic of each fault and minimum the Bayes risk of misdiagnosis (Hellman and Raviv, 1970; Tawarmalani and Sahinidis, 2002; Zhu et al., 2024). For discrete-time stochastic systems, a new upper bound of misdiagnosis probability for incipient actuator faults is proposed by Guo and He (2024), which promotes the diagnosis performance.

The set-based method describes different faults by sets. To achieve the purpose of fault diagnosis, this method designs an input so that there is no intersection between the sets of systems (Andjelkovic and Campbell, 2011; Nikoukhah, 1998; Scott and Campbell, 2014; Tabatabaeipour, 2015). The AFD method introduced by Xu (2021) addresses a two-level optimization problem by designing a single input at each time step. In a study by Xu (2024), a newly defined excluding degree of the origin from all healthy and faulty residual zonotopes is proposed, which owns more explicit physical meaning.

While the aforementioned studies have made notable progress, these methods are restricted for nonlinear systems due to computational challenges and the complexity of auxiliary signal design. In the past decade, data-driven methods have gained widespread popularity and proven effective in addressing various nonlinear problems (Song et al., 2023). One of these methods is deep reinforcement learning (DRL).

In essence, DRL can be viewed as an optimization algorithm. The control objective is transformed into an optimization function, and the optimal solution of this function is the control signal (Silver et al., 2014; Van Hasselt et al., 2016; Wang et al., 2016; Yan and Xu, 2019). DRL has demonstrated effectiveness across various domains (Wang et al., 2024). For example, some researchers have applied the DRL to nonlinear systems such as fuzzy systems and switched systems, which achieved prescribed time convergence (Zhang et al., 2024a; 2024b; Zhang and Xiang, 2025). In recent years, several studies (Li et al., 2023; Wang et al., 2023b) have integrated DRL into fault diagnosis with promising results. However, these studies mostly focus primarily on fault diagnosis assumptions while ignoring the impact of input signals on system performance. The main reason is that the research of fault diagnosis with DRL is not mature enough. Moreover, the conflict between fault diagnosis and control performance poses challenges in designing reward functions within the DRL framework. A study referenced as literature (Yan et al., 2023) introduces an Actor-Critic (AC) framework which is a well-received method in DRL. The research transforms the tracking issue into a constrained optimization problem and leverages the Trust Region Policy Optimization (TRPO) method to address the auxiliary signal design aspects of AFD. To evaluate the efficacy of this methodology, the study conducts two case studies, demonstrating the application of this approach in practical scenarios.

Based on the effort of the literature above, we propose a novel AC framework to generate the tracking and diagnostic signals respectively. Furthermore, to deal with the tradeoff between tracking performance and fault diagnosis, we present a two-objective optimization method based on deep reinforcement learning.

The implementation of this approach adopts a dual AC architecture. One AC architecture is responsible for fault diagnosis, its critic network assesses the accuracy of fault diagnosis, and its actor network generates fault diagnosis inputs. The Other AC architecture focuses on optimizing tracking performance, its critic network evaluates system controlling performance, and its actor network outputs control inputs.

The balance of control and diagnostic performance is reflected in the design of the reward function. A tradeoff parameter is introduced to quantify the balance between control and diagnosis. The parameter can be flexiblely adjusted to satisfy different requirements. In a study by Jia et al. (2023), an optimal trajectory is generated to diagnose faults, and the controller is designed to track this trajectory. Actually, the design of the controller depends on the dynamics, different from Jia et al. (2023), the method proposed in this paper adopts a model-free strategy, where the controller’s design is not reliant on system dynamics but instead on a large quantity of data to achieve a balance between control and diagnosis requirements. Similarly, in a study by Wang et al. (2023a), the utilization of the Q-learning method serves as an aid to control algorithms, rather than employing a completely model-free strategy. By adopting a model-free method, complex mathematical derivation can be avoided, rendering the design more intuitive and comprehensible. To facilitate computational efficiency, we employ Proximal Policy Optimization (PPO) as the policy learning method, which simplifies the optimization problem encountered in the TRPO algorithm. This choice not only streamlines the calculation process but also promotes more stable convergence due to the limitations imposed on gradient updates.

In general, the main contributions are as follows:

An Active Fault Diagnosis Tracking (AFDT) network is introduced in this paper, the network can output the auxiliary signal for AFD and ensure the control performance of the nonlinear system. The nonlinear system can be unknown, it depends on little prior knowledge.

In contrast to the auxiliary signal design by Li et al. (2023) and Wang et al. (2023b), the auxiliary signal design in this paper can guarantee the control performance while the fault diagnosis being conducted.

Compared with Yan et al. (2023), the compromise scheme between diagnostic accuracy and control performance is solved by a deep reinforcement learning method. Furthermore, to mitigate computational expenses, the PPO method is implemented as a replacement for TRPO, ensuring algorithmic stability.

The rest of the content of this paper is as follows: Section 2 introduces the system model and the background knowledge about RL. Section 3 contains the design of the AFD network and the main problem of AFD with control constraints. A simulation of the cart-pole system is presented in Section 4 to show the effect of the AFDT network. The conclusion is located in Section 5.

Problem statements

System modeling and problem formulation

Assume there exists a series of discrete-time nonlinear stochastic system, which is described as follow (Yan et al., 2023):

F^{i} : {\begin{matrix} x_{k + 1}^{i} & = f (x_{k}^{i}, u_{k}, H_{i}, w_{k}) \\ y_{k}^{i} & = g (x_{k}^{i}, u_{k}, H_{i}, v_{k}) \end{matrix}, i \in I ≜ {1, \dots, n_{f}}

(1)

where i indicates the $F^{i}$ system with fault i and the number of fault types is $n_{f}$ . $x_{k}^{i} \in ℝ^{n_{x}}, y_{k}^{i} \in ℝ^{n_{y}}, u_{k} \in ℝ^{n_{u}}$ represents the state, output and input of system $F_{i}$ , respectively. The matrix $H_{i} \in ℝ^{n_{u} \times n_{u}}$ denotes the actuator fault matrix of the system, which acts on the input and corresponds to $F^{i}$ system one by one. $w_{k} \in ℝ^{n_{w}}$ and $v_{k} \in ℝ^{n_{v}}$ represent the noise of the system, which can be disturbances and measurement noise. Both of them follow independent Gaussian distributions, i.e., $w_{k} \sim N (μ_{w}, σ_{w})$ and $v_{k} \sim N (μ_{v}, σ_{v})$ . Functions $f (\cdot)$ and $g (\cdot)$ are the nonlinear system dynamics and output, respectively.

With the assumption above, the fault matrix $H_{i}$ in equation (1) is defined as:

H_{i} ≜ diag (f_{1}^{i}, f_{2}^{i}, \dots, f_{n_{u}}^{i}), i \in I

(2)

where $f_{j}^{i} \in (0, 1), j \in {1, 2, \dots, n_{u}}$ indicates the performance loss of the jth actuator. A series of $f_{j}^{i}, j \in {1, 2, \dots, n_{u}}$ represent a type of fault, there are $n_{f}$ kinds of faults in total. Particularly, we define $F^{0}$ as the system without fault, and the corresponding fault matrix is $H_{0} = I$ .

For auxiliary control signal design, a reference system is defined to provide a reference:

{\begin{matrix} x_{k + 1}^{ref} & = f (x_{k}^{ref}, u_{k}^{ref}, H_{0}) \\ y_{k}^{ref} & = g (x_{k}^{ref}, u_{k}^{ref}, H_{0}) \end{matrix}

(3)

where $x_{k}^{ref}, y_{k}^{ref}$ are the reference state and the reference output of system with the same dimention in system (1), respectively. Input $u_{k}^{ref}$ denotes an independent bounded input signal which stabilizes the system (3). All the initial states $x_{0}^{i}$ in system $F^{i}, i \in I ⋃ {0}$ are the same and follow the Gaussian distribution. For the fault system $F^{i}$ , the state $x_{k}^{i}$ and output $x_{k}^{i}, y_{k}^{i}$ of different fault systems are independent. However, the inputs $u_{k}$ are shared.

We assume $D (y_{1}, y_{2})$ to be a measurement that calculates the distance between vector $y_{1}$ and $y_{2}$ , in this paper, we choose L1 norm $D (y_{1}, y_{2}) = | | y_{1} - y_{2} | |_{1}$ . For the system (1) and (3), the AFD with tracking constraint can be described as: with the error between output signal $y_{k}$ and reference signal $y_{k}^{ref}$ being less than β, designing an input $u \in U$ , such that the distance among the output of different kinds of faults system is max.

It can be denoted as:

\begin{matrix} \max_{u \sim π_{θ} \in U} \sum_{i = 1}^{n_{f} - 1} \sum_{j = i + 1}^{n_{f}} D (y_{k}^{i}, y_{k}^{j}) \\ s . t . | | y_{k} - y_{k}^{ref} | |_{1} \leq β \end{matrix}

(4)

where β is the threshold of tracking error.

Reinforcement learning preliminaries

Assume there exists an agent follows a Markov Decision Process (MDP), which consist of tuple $T = (S, A, P, R, γ)$ , in which:

$S$ is the state space, and the state $s_{t} \in S$ represents the state of agent at instant t

$A$ is the action space, the action at instant t is defined as $a_{t} = π (a_{t} | s_{t}) ≜ \Pr (A_{t} = a_{t} | S_{t} = s_{t})$ , the distribution π is called policy.

$R : S \times A \times S \to ℝ$ , is the reward, where $R (s_{t + 1}, a_{t}, s_{t})$ is the reward when state $s_{t}$ takes action $a_{t}$ with the next state being $s_{t + 1}$ .

$P : S \times A \times S \to ℝ$ , is the transition probability, where $P (s_{t + 1}, a_{t}, s_{t})$ is the transition probability of state $s_{t}$ takes action $a_{t}$ with the next state being $s_{t + 1}$ .

γ is a discounted parameter, $γ \in [0, 1]$ .

The process that the agent changes from state $s_{t}$ to $s_{t + 1}$ at some instant is called a transition, and tuple $(s_{t}, a_{t}, r_{t}, s_{t + 1})$ is used to represent a transition. A transition of RL is presented in Figure 1.

Figure 1.

The RL process.

After several transitions, the agent could reach a terminated state at instant T (which means the target is accomplished or not), generating a trajectory described as $(s_{0}, a_{0}, r_{0}, s_{1}, \dots, s_{T - 1}, a_{T - 1}, r_{T - 1}, s_{T})$ . Then, in this trajectory, the total reward of state $s_{t}$ is defined as:

V_{π} (s_{t}) = (r_{0} + γ r_{1} + γ^{2} r_{2} + \dots + γ^{T} r_{T}) = \sum_{i = 0}^{T} γ^{i} r_{i}

(5)

Generally speaking, the reward $r_{t}$ at instant t differs among different tasks. For example, the reward for the problem (4) can be formulated as $r_{t} = 0.5 \sum_{i = 1}^{n_{f} - 1} \sum_{j = i + 1}^{n_{f}} | | y_{t}^{i} - y_{t}^{j} | | - 0.5 | | y_{t} - y_{t}^{ref} | |$ , which takes both the tracking target and the AFD target into consideration. Obviously, the total reward represents the value of state $s_{t}$ to a certain extent, but the reward above is random, which makes $V_{π} (s_{t})$ not stable enough. Thus, taking the expectation of total reward, the state value function $V_{π} (s_{t})$ can be presented as:

V_{π} (s_{t}) = E_{a_{i} \sim π (\cdot | s_{t}), s_{t + 1} \sim P (\cdot | s_{t}, a_{t})} [\sum_{i = 0}^{\infty} γ^{i} R_{i}]

(6)

Similarly, the action value function is defined, which represents the value of action $a_{t}$ in state $s_{t}$ .

Q_{π} (s_{t}, a_{t}) = E_{s_{t + 1}, a_{t + 1}, \dots} [\sum_{i = 0}^{\infty} γ^{i} R_{i}]

(7)

On the basis of $V_{π} (s_{t})$ and $Q_{π} (s_{t}, a_{t})$ , advantage function is defined as:

A_{π} (s_{t}, a_{t}) = Q_{π} (s_{t}, a_{t}) - V_{π} (s_{t})

(8)

which denotes the advantage of action $a_{t}$ over all possible actions under the policy $π (\cdot | s_{t})$ . For RL, the higher total reward means that the agent is more likely to achieve the goal, so the optimization target of RL is to find a suitable policy to maximize $V_{π_{θ}} (S)$ . It can be derived as:

π^{*} = \underset{π}{\arg \max} (E_{S} [V_{π} (S)])

(9)

By sampling the target policy $π^{*}$ , the systems (1) and (3) get their inputs. In DRL, the policy can be fitted by a neural network $π (\cdot | s_{t}; θ)$ , where θ is the parameter of the network. To make $V_{π}$ as large as possible, Policy Gradient (PG) ascent is used. It can be proven that PG can converge to a target policy $π^{*}$ (Sutton et al., 1999), and the PG ascent can be derived as:

θ^{'} \leftarrow θ + α \nabla_{θ} V_{π_{θ}}

(10)

where α is the learning rate, which regulates the pace of gradient ascent. In actual training, the selection of α is a challenging task. If α is too large, excessive variation will occur during parameter updating, which easily makes the convergence curve unstable. Conversely, a very small α will lead to slow convergence and increase calculation costs.

TRPO method is proposed to solve this problem, by imposing a constraint on the Kullbac–Leibler (KL) divergence between the new policy and old policy. These two policies are close enough to use the importance sampling theorem with minor variance.

In equation (11), $V_{π} (s)$ is transformed from an expectation about the policy $π (\cdot | s_{t}; θ)$ to an expectation about the new policy $π (\cdot | s_{t}; θ_{new})$ .

\begin{matrix} V_{π} (s_{t}) & = \sum_{a_{t} \in A} π (a_{t} | s_{t}; θ) Q_{π} (s_{t}, a_{t}) \\ = \sum_{a_{t} \in A} π (a_{t} | s_{t}; θ_{new}) \frac{π (a_{t} | s_{t}; θ)}{π (a_{t} | s_{t}; θ_{new})} Q_{π} (s_{t}, a_{t}) \\ = E_{a_{t} \sim π (\cdot | s_{t}; θ_{new})} [\frac{π (a_{t} | s_{t}; θ)}{π (a_{t} | s_{t}; θ_{new})} Q_{π} (s_{t}, a_{t})] \\ = W (θ | θ_{new}) \end{matrix}

(11)

Then the gradient ascent problem is transformed into a optimization problem by TRPO (Schulman et al., 2015):

\begin{matrix} \underset{θ_{t + 1}}{\arg \max} (W (θ_{t} | θ_{t + 1})) \\ s . t . {\bar{D}}_{KL} (π_{θ_{t}} ∥ π_{θ_{t + 1}}) \leq δ \end{matrix}

(12)

where δ is a threshold to limit the KL divergence.

Although TRPO reduces the dependence of PG on the learning rate, it takes large computational consumption. Thus, the PPO method is proposed to simplify it, and the constraint on KL divergence is integrated into the optimization function. The target policy is as follows (Schulman et al., 2017):

π_{θ^{*}}^{*} = \underset{π_{θ}}{\arg \max} {E_{t} [\min (r_{t} (θ) A_{π_{θ}}, clip (r_{t} (θ), 1 - ϵ, 1 + ϵ) A_{π_{θ}})]}

(13)

where $r_{t} (θ) = π_{θ} (a_{t} | s_{t}) ∕ π_{θ_{old}} (a_{t} | s_{t})$ , ϵ is the clipped factor, $π_{θ_{old}}$ means the policy which is not updated at instant t, $A_{π_{θ}}$ denotes the advantage function. The PPO method incorporates a clip function to restrict the ratio between the new policy and old policy within a specified range. This constraint ensures that the updated policy remains close to the original policy and effectively limits the magnitude of the policy gradient updates.

Main results

In this section, a DRL-based AFD method with controlling performance guaranteed for nonlinear systems is presented. A double actor-critic network structure is introduced to generate a controlling signal and an AFD signal. A fitting network integrates these two signals to get the input of the detected system. The overview of whole network structure can be depicted in Figure 2.

Figure 2.

AFD with tracking performance guaranteed.

Network structure

According to the problem statement, the input can be divided into two parts. One is to ensure the performance of controlling, which is realized by tracking the reference signal, thus we assume it to be $u_{1} \in U_{tracker}$ , and the other is to maximize the distance among different fault systems’ outputs, which is assumed to be $u_{2} \in U_{AFD}$ . The inputs $u_{1}$ and $u_{2}$ are generated by two actor-critic networks, respectively. Finally, the input of the detected system will be produced by a fitting network whose inputs are $u_{1}$ and $u_{2}$ . The fitting network can be derived as $u = h (u_{1}, u_{2}; λ)$ , where λ is the parameter of the fitting network. Next, the networks above will be introduced in detail.

Tracker network

The tracker network has an AC structure, the actor network outputs the tracking signal for the detected system. The critic network outputs the value of state $s_{t}$ . Assuming the critic network to be $V_{π} (s_{t}; ϕ)$ , the actor network to be $π (\cdot | s_{t}; θ)$ , where ϕ and θ are both parameters of networks, respectively. Normally, the state of the tracker network is set as:

s_{t} = [x_{t}, x_{t}^{ref}, y_{t}, y_{t}^{ref}]

(14)

In the critic network, a reward function is involved to measure the tracking effectiveness of the detected system. The reward function can be written as:

c_{t} (s_{t}, a_{t}) = k_{1} d_{1} + 1

(15)

where $d_{1} = | | ẏ_{t} - ẏ_{t}^{ref} | |_{1}$ , $k_{1} < 0$ is a reward adjusting parameter set manually. With this linear map, the maximal reward is limited to 1, which is beneficial to train network because it will not cause big numerical fluctuation.

Remark 1. A brief analysis of the optimal function is listed here: By designing the reward function as specified in equation 15, the value function can be expressed as: $V_{π} (s) = \sum_{i = 0}^{T} γ^{i} (k_{1} d_{1} (i) + 1)$ , assuming the sampling time is sufficiently small, the value function can be appropriated as: $\int_{0}^{T} γ^{t} (k_{1} d_{1} (t) + 1) dt$ . Here, the constant term, which functions as an offset, can be disregarded. The integral of the remaining term is:

\begin{matrix} \int_{0}^{T} γ^{t} k_{1} d_{1} (t) dt \\ = γ^{t} k_{1} | | y_{t} - y_{t}^{ref} | |_{1} |_{0}^{T} - \int_{0}^{T} k_{1} | | y_{t} - y_{t}^{ref} | |_{1} γ^{t} lnγ dt \\ \approx k_{1} | | y_{T} - y_{T}^{ref} | |_{1} γ^{T} \end{matrix}

The term $\int_{0}^{T} k_{1} | | y_{t} - y_{t}^{ref} | |_{1} γ^{t} lnγ dt$ is neglected since γ is approximately equal to 1. This analysis demonstrates that the value function associated with the reward in equation 15 corresponds to the norm value of $y_{T}$ and $y_{T}^{ref}$ , thus the optimal policy from this value function minimizes the error between $y_{T}$ and $y_{T}^{ref}$ .

By calculating the total reward $C_{π} (s_{t}) = \sum_{i = 0}^{\infty} (γ^{i} c_{t + i})$ , the tracker network adjusts policy to follow the reference system. Naturally, the loss function of the critic network $V_{π} (s_{t}; ϕ)$ is:

L (ϕ) = E_{s_{t + 1} \sim p (\cdot | s_{t}, a_{t}), a_{t} \sim π (\cdot | s_{t})} [0.5 {(C_{t} - V_{π} (s_{t}; ϕ))}^{2}]

(16)

where $C_{t}$ is the real reward from the environment.

In the action network, we employ a continuous random policy of beta distribution instead of a discrete policy to ensure control accuracy. The outputs of the actor network are two parameters of beta distribution ( $α, β$ ). The agent can sample the next action according to this beta distribution.

Based on the loss function in equation (16) and the PPO method in equation (13), let $I (θ) = E_{t} [\min (r_{t} (θ) A_{π_{θ}}, clip (r_{t} (θ),$ $1 - ϵ, 1 + ϵ) A_{π_{θ}})]$ , the updates of parameters ϕ and θ are:

\begin{matrix} θ^{'} \leftarrow θ + α_{1} \nabla_{θ} I (θ) \\ ϕ^{'} \leftarrow ϕ - α_{2} \nabla_{ϕ} L (ϕ) \end{matrix}

(17)

In many cases, the function $Q_{π} (s_{t}, a_{t})$ is unknown, thus, it needs to be estimated. Monte Carlo (MC) theorem provides a method: the Q value can be estimated by using the rewards in a trajectory (Williams, 1992). Since these rewards follow the distribution $R (s_{t + 1}, a_{t}, s_{t})$ , the MC estimation is unbiased, it can be used in the calculation of policy gradient.

AFD network

The AFD network is of the same structure as the tracker network, but the reward of the AFD network is associated with fault diagnosis. In consideration of the problem stated in equation (4), using similar linear map method in equation (15), it is easy to get the following formulation:

r_{t} (s_{t}, a_{t}) = k_{2} d_{2} - 1

(18)

where $d_{2} = \sum_{i = 1}^{n_{f} - 1} \sum_{j = i + 1}^{n_{f}} | | {\overset{\cdot}{y_{t}}}^{i} - {\overset{\cdot}{y_{t}}}^{j} | |_{1}$ and $k_{2} > 0$ is a reward adjusting parameter set manually.

Remark 2. The reward function here follows a formulation similar to that in equation 15. By applying the same method in remark 1, it can be concluded that the value function corresponds to the norm of $y_{T}^{i}$ and $y_{T}^{j}$ . Therefore, the optimal policy is capable of distinguishing between the outputs of different fault systems.

It should be pointed out that the reward in equation (18) has a fatal problem, which is that the reward has no diagnosis specificity for each fault. For example, assuming that a designed input can diagnose two faults exceptionally well, but diagnose others poorly. This designed input can also produce a large reward, but it is not expected. To avoid this situation and get a better fault diagnosis effect, the maximum mean discrepancy (MMD) method is involved.

MMD is a principle to measure the similarity of two distributions. Assuming there exists two sets of observed samples $X = {x_{1}, x_{2}, \dots, x_{m}}$ and $Y = {y_{1}, y_{2}, \dots, y_{n}}$ , which are independently and identically distributed from p and q, respectively. The MMD of two sets can be defined as Schölkopf et al. (2007):

MMD [F, p, q] = \sup_{f \in F} (E_{x \sim p} [f (x)] - E_{y \sim q} [f (y)])

(19)

where f is a map function from the function space $F$ , and the space $F$ is a unit-ball in a Reproducing Kernel Hilbert Space (RKHS). Because only partial samples have been observed for both distributions, empirical estimates are needed here:

MMD [F, p, q] = \sup_{f \in F} (\frac{1}{m} \sum_{i = 1}^{m} [f (x_{i})] - \frac{1}{n} \sum_{i = 1}^{n} [f (y_{i})])

(20)

By using the trick of the kernel function, MMD can be calculated without knowing the function f (both the expected form and the empirical estimate form are given):

\begin{matrix} {MMD}^{2} [F, p, q] = ∥ E_{x \sim p} [k (x, \cdot)] - E_{y \sim q} [k (y, \cdot)] ∥^{2} \\ {MMD}^{2} [F, X, Y] = {∥ \frac{1}{m} \sum_{i = 1}^{m} k (x_{i}, \cdot) - \frac{1}{n} \sum_{j = 1}^{n} k (y_{j}, \cdot) ∥}^{2} \end{matrix}

(21)

where $k (\cdot, \cdot)$ indicates the kernel function. Usually, the Gaussian kernel function is used over a vectorized representation of N-step trajectories.

K (ϕ_{N} (τ), ϕ_{N} (τ^{'})) = \exp (- \frac{{∥ ϕ_{N} (τ) - ϕ_{N} (τ^{'}) ∥}_{2}^{2}}{2 σ^{2}})

(22)

where $ϕ_{N} (τ)$ is a vector including all the historical data.

Under the consideration of MMD, the input of the AFD network is derived as:

\begin{matrix} s_{t} = [ & l^{1} x_{t}^{1}, \dots, l^{n_{f}} x_{t}^{n_{f}}, \\ l^{1} y_{t}^{1}, \dots, l^{n_{f}} y_{t}^{n_{f}}, l^{1}, \dots, l^{n_{f}}] \end{matrix}

(23)

where $x_{t}^{i}, y_{t}^{i} (i = 1, 2, \dots, n_{f})$ are the states and outputs of fault systems, $x_{t}, y_{t}$ are the state and output of the detected system. $l^{i} (i = 1, 2, \dots, n_{f})$ is a boolean variable to assist fault diagnosis with MMD. The reward is rewritten as:

r_{t} (s_{t}, a_{t}) = - b_{2} + k_{2} \sum_{i = 1}^{n_{f} - 1} \sum_{j = i + 1}^{n_{f}} l_{i} l_{j} | | {\overset{\cdot}{y_{t}}}^{i} - {\overset{\cdot}{y_{t}}}^{j} | |_{1}

(24)

where $k_{2}, b_{2} > 0$ are the linear map parameters.

Remark 3. Generally speaking, if ${MMD}^{2} [F, p, q] = 0$ , it indicates the distributions p and q are the same. Thus, the MMD index serves to identify the least likely fault mode. Through successive iterations, this index aids in excluding the least probable fault mode. Subsequently, by setting the boolean variable $l_{i}$ as FALSE, redundant states can be effectively eliminated.

Remark 4. This reward function is a variant of equation 18, the key different between equations 18 and 24 is the involvement of MMD, actually, the reward in equation 24 just masks some distance between system outputs dynamically.

Next, the algorithm of the AFD network with MMD is introduced.

First, set all l to TRUE and initialize the systems’ parameters. Then, calculate the MMD between the detected system and fault systems after some time. Next, find the fault system corresponding to the largest MMD and exclude this kind of fault. Finally, repeat this process for the remaining fault systems until only one flag remains TRUE. The fault type corresponding to this remaining flag is the result. The total algorithm is presented in algorithm 1.

Algorithm 1 Training of AFD network
Input: initial state $x_{0}$ ; diagnostic sequence length $len$ ; $n_{f}$ fault systems; current system; $l^{i} (i = 1, 2, \dots, n_{f}) = 1$ ; step = 1.
while $\sum_{i = 1}^{n_{f}} > 1$ :
Input $x_{t}$ into AFD network and get the action $a_{t}$
if mod(step, $len$ ) = 0 then
Calculate the MMD of each fault system and the detected system, get $n_{f}$ MMD.
Find the fault system corresponding to the maximum MMD, set the corresponding $l^{i}$ to 0.
endif
step = step + 1.
end

Whole algorithm

The outputs of the AFD network and tracker network are loaded into the fitting network. In order to improve training efficiency, the AFD network and tracker network are trained offline.

During the training process of the whole network, the parameters in AFD and tracker networks are not updated. The only updated parameters are in the fitting network, they are updated by using the policy gradient ascent method, which is shown in equation (10). Since $V_{π} (s)$ is difficult to get in practice, it is proven that the gradient of $V_{π} (s)$ has the equivalent form (Sutton et al., 1999):

\nabla_{θ} V_{π_{θ}} = \nabla_{θ} lnπ (a | s; θ) Q_{π} (s, a)

(25)

In equation (25), $lnπ (a | s)$ is known, and $Q_{π} (s, a)$ can be estimated in a trajectory according to the MC theorem.

Considering both tracking and diagnostic performance, the reward function of the fitting network is designed as:

\begin{matrix} r_{t}^{'} = (1 - λ) r_{1}^{'} + λ r_{2}^{'} \\ r_{1}^{'} = - 1 + k_{3}^{1} \sum_{i = 1}^{n_{f} - 1} \sum_{j = i + 1}^{n_{f}} ∥ y_{t}^{i} - y_{t}^{j} ∥_{1} \\ r_{2}^{'} = - b_{3} + k_{3}^{2} ∥ \overset{\cdot}{y_{t}} - {\overset{\cdot}{y_{t}}}^{ref} ∥_{1} \end{matrix}

(26)

where $λ \in (0, 1)$ is the weight parameter, adjusting diagnosis and tracking reward focus. $k_{3}^{1}, k_{3}^{2}$ and $b_{3} > 0$ are the linear map parameters. The training process of the fitting network is depicted in algorithm 2.

Remark 5. As depicted in Figure 2, there is a shared dense layer responsible for generating the initial features. Two separate networks are employed to produce the AFD signal and control signal, which are devoted to seeking optimum solutions for their respective optimization problems. The fitting network comes into play to harmonize the tracking and AFD performances, with the parameter λ serving as a balance to adjust preferences between control performance and fault diagnosis performance.

Remark 6. The article addresses a common challenge known as the ’seesaw problem’, where the AFD signal’s impact on tracking performance and vice versa. The primary concept presented involves transforming the dual-objective optimization into a singular objective by employing a weighted sum technique. In this approach, the tracker network and the AFD network are employed to generate two boundary Pareto solutions. To convert the ’seesaw problem’ into a singular-objective optimization, the weighting parameter λ and the fitting network are introduced. The reward function applied to train the fitting network consists of a weighted sum of two rewards (See equation 26). The weighted reward function represents the optimization preference. From the perspective of the Pareto solution, the fitting network is to find a mean solution in the Pareto solution set.

Algorithm 2 Training of the Fitting network
Input: AFD network; tracker network; initial state $x_{0}$ ; λ; update time; batch size; training trajectory; cyclic variable $i = j = 0$
while $i <$ training trajectory:
Sample actions following policy $π (\cdot \| s_{t}; θ)$ get several transitions $(s_{t}, a_{t}, r_{t}, s_{t + 1})$ , save them into buffer $M$
while $j <$ updatetime:
Sample from buffer $M$ for batchsize times
Compute Q value by using reward in buffer $M$
Get the policy gradient via equation (25)
Update the parameters in fitting network, but don’t update the AFD and tracking networks
end
Empty buffer
end

The training process of these three networks is conducted offline. The trained networks are loaded onto the upper computer when fault diagnosis is needed. Therefore, the online computational cost is focused on the networks’ complexity. Before calculating the computation costs, some preliminaries should be stated:

Generally speaking, the calculation of the neural network can be regarded as:

y = W_{n_{2} \times n_{1}} {[x_{1}, x_{2}, \dots, x_{n_{1}}]}^{T} + b_{n_{2} \times 1}

(27)

where y represents the output, $[x_{1}, x_{2}, \dots, x_{n_{1}}]$ denotes the input vector, matrix W comprises the network parameters, $b_{n_{1} \times 1}$ stands for the bias term.

In equation 27, there are:

$[n_{1} \times$ multiplications + $(n_{1} - 1) \times$ additions $] \times n_{2} + n_{2} \times$ additions.

After sorting, the neural network in equation 27 performs $n_{1} n_{2}$ multiplications and additions. Assuming the activation function is $\tanh (\cdot)$ and employing the Taylor expansion equation:

\tanh (x) = \sum_{n = 1}^{\infty} \frac{2^{2 n} (2 n - 1) B_{2 n} x^{2 n - 1}}{(2 n)!}

(28)

Expand the Taylor series to the second term, $\tanh (x)$ necessitates at least 5 multiplications and 1 addition. Consequently, one layer of the neural network consumes $(5 n_{2} + n_{1} n_{2})$ multiplications and $(n_{2} + n_{1} n_{2})$ additions in total.

Simulation

Training setup

The AFD and tracking networks are of the same AC network structure, the critic network is made up of three hidden layers, with hidden units being 64, 64, and 32, respectively. The squashing functions among these layers are $\tanh (\cdot)$ . The actor network consists of one shared hidden layer and two separate sets of hidden layers that fit the α and β parameters, respectively. The shared hidden layer is of 64 hidden units with the squashing function $\tanh (\cdot)$ followed. The separate sets of hidden layers are constituted by two hidden layers, the hidden layers are of 64 hidden units, with the squashing function being $\tanh (\cdot)$ and $softplus (\cdot)$ . Given that most activation functions are now $\tanh (\cdot)$ , for simplicity, the activation function calculations between each layer align with $\tanh (\cdot)$ . The critic network needs:

( $15 \times 64 + 69 \times 64 + 69 \times 32 + 32 = 7616$ ) multiplications

( $11 \times 64 + 65 \times 64 + 65 \times 32 + 32 = 6976$ ) addtions.

The actor network needs:

[ $15 \times 64 + 2 \times (69 \times 64 + 69 \times 64 + 64) = 18752$ ] multiplications

[ $11 \times 64 + 2 \times (65 \times 64 + 65 \times 64 + 64) = 17472$ ] addtions.

To sum up, the AFD and tracking networks each take ( $7616 + 18752 = 26, 368$ ) multiplications and ( $6976 + 17472 = 24, 448$ ) additions.

The fitting network is of the same structure as the actor network, but the shared hidden layer is of 256 hidden units with input dimention being two times of the actor network output dimention. Thetwo separate sets of hidden layers are both composed of three hidden layers, with hidden units being 256, 128, and 64, respectively. The squashing functions among them are $\tanh (\cdot)$ , $\tanh (\cdot)$ and $softplus (\cdot)$ . Similarly, the fitting network takes 219,328 multiplications and 214,720 additions in total.

During the offline training, the AFD, tracker, and fitting networks are trained separately. For linear map parameters, $k_{1} = - 5.0$ and $k_{3}^{1} = - 2.0$ , the other linear parameters $k_{2}, b_{2}, k_{3}^{2}$ , and $b_{3}$ follow the relationship in equation (29)

{\begin{matrix} k = 1 ∕ 0.3, b = 1 & if l_{1} + l_{2} + l_{3} = 3 \\ k = 1 ∕ 0.4, b = 1 & if l_{1} + l_{2} + l_{3} = 2 \\ k do n^{'} t care, b = 0 & if l_{1} + l_{2} + l_{3} = 1 \end{matrix}

(29)

The other main training parameters are listed in Table 1. α is learning rate, γ is the discount factor of value function, ϵ is the clipped factor in PPO, λ is the weight parameter in the fitting network reward function. The terminating step means the maximum transition number in every trajectory.

Remark 7. Here, some suggestions about the parameters design are given.

Parameter γ indicates the influence range of an action, where γ is closer to 1 means that the more influence on the future states.

Parameter ϵ is used to limit the update step, which avoids the influence of large gradient updates on the stability of the training process. Commonly, it is chosen as 0.2.

Parameter λ is to balance the performance of tracking and AFD. It is suggested that $λ > 0.5$ . Fo the reason that the AFD network always leads to a bad tracking performance, we need to concentrate more on tracking performance.

Table 1.

Training parameters.

Parameters	AFD	Tracker	Fitting
$α_{0} (PG)$	–	–	$5 \times 1 0^{- 6}$
$α_{1} (actor)$	$5 \times 1 0^{- 5}$	$5 \times 1 0^{- 5}$	–
$α_{2} (critic)$	$1 0^{- 4}$	$1 0^{- 4}$	–
γ	0.85	0.80	0.80
ϵ	0.20	0.20	–
λ	–	–	0.80
Batch size	64	128	64
Update time	3	3	15
Terminating steps	100	1000	500

Training case

The trainging case of this method is a cart-pole system, the details of modeling can be found in a study by Barto et al. (1983). Assuming there is a cart moving on a horizontal smooth surface, its mass is M. A pole of length $2 l$ and mass m is attached to the cart center. Suppose that any contact point is completely smooth, take the central position of the cart on the X-axis as the origin O, specify the positive direction to the right, the angle between the center line of the connecting pole and the vertical direction as θ, and take the clockwise direction as the positive direction. The connecting pole is kept vertical by controlling the force applied to the cart in the horizontal direction.

Let $z = {[x_{t}, \overset{\cdot}{x_{t}}, θ_{t}, \overset{\cdot}{θ_{t}}]}^{T}$ , and the discretized dynamic model can be equation (30):

\begin{matrix} z_{1} (k + 1) = z_{1} (k) + Δ T z_{2} (k) \\ z_{2} (k + 1) \\ = z_{2} (k) + \frac{Δ T (u (k) + ml [{\overset{\cdot}{z_{3}}}^{2} (k) \sin z_{3} (k) - \overset{··}{z_{3}} (k) \cos z_{3} (k)])}{M + m} \\ z_{3} (k + 1) = z_{3} (k) + Δ T z_{4} (k) \\ z_{4} (k + 1) \\ = z_{4} (k) + \frac{Δ T (gsin z_{3} (k) - \cos z_{3} (k) \frac{u (k) + ml {\overset{\cdot}{z_{3}}}^{2} (k) \sin z_{3} (k)}{M + m})}{l (\frac{4}{3} - \frac{mco s^{2} z_{3} (k)}{M + m})} \end{matrix}

(30)

During the training process, there are two criteria that determine the termination of a trajectory. First, if the angle θ or the displacement x exceeds a predefined threshold, it signifies a termination state. In this case, it indicates that the controller has failed to achieve the desired control target. Second, if the cart-pole system reaches the terminating steps, it is considered that the control target has been successfully achieved. These criteria serve as the indicators for determining the end of a trajectory during training.

In this paper, we assume that $m = 0.1 kg, M = 1 kg, l = 0.5 m$ . Besides, the angle threshold is $\pm 0.418 rad$ , the displacement threshold is $\pm 4.8 m$ , sampling time $Δ T = 0.005 s$ , the noise of the system are $w_{k} \sim N (0, 1 0^{- 4})$ and $v_{k} \sim N (0, 1 0^{- 3})$ . The faults in this paper are assumed to be multiplicative faults of the actuator, acting on the input $u (k)$ . In this paper, the main faults are divided into three types, respectively:

\begin{matrix} G_{1} = N (0.20, 0.04) \\ G_{2} = N (0.50, 0.04) \\ G_{3} = N (0.80, 0.04) \end{matrix}

(31)

The parameters $G_{1}, G_{2}$ and $G_{3}$ will be multiplied by the input $u (k)$ , respectively.

The reference system is of the same model in equation (30), stabilized by PID controller with the controller parameters being $K_{p} = 20, K_{i} = 0.01$ and $K_{d} = 8$ . The tracker network will follow the output of the reference system to guarantee the performance of the system to some extent.

Smiulation and discussions

In this paper, the training of the tracker network takes 3000 trajectories, and the training of the AFD network takes 5000 trajectories. A learning curve of the tracker network and AFD network are given in Figure 3 to present the convergence of these two networks.

Figure 3.

Rewards of AFD network and tracker network: (a) AFD reward. (b) Tracker reward.

According to the reward defined in equation (15), the maximum total reward is 1000, with each step contributing a reward of 1 over a trajectory of 1000 steps. The observed convergence value is approximately 940. While the maximum total reward represents an ideal outcome, minor deviations are unavoidable. Therefore, we can conclude that the network has successfully completed the tracking task. In addition, a test was conducted to evaluate the performance of the tracker network. Figure 4 illustrates the results, where the actual states of two systems and the corresponding errors between them are plotted on a graph with dual axes. It can be observed that the vertical angle error remains consistently below 0.1 rad, approaching zero. Furthermore, the displacement of the cart remains within the defined threshold over 1000 steps, indicating that the tracking network effectively achieves the desired controlling objective.

Figure 4.

Tracking performance of tracker network: (a) Angle tracking effect, (b) Palstance tracking effect, (c) Displacement tracking effect and (d) Velocity tracking effect.

For the AFD networks, since no upper limit is given in the reward setting, it is difficult to judge the maximum value of the total reward. However, it can be observed from the figure that the learning curve eventually converges. Considering that the main purpose of the AFD network is to maximize the separation of the outputs between the various faulty systems, so the actual capability of the network can be detected by testing, Figure 5 shows the AFD network performance of fault separation, it can be seen in the figure that the AFD network separates each fault, which basically realizes the task of the AFD network. Admittedly, the AFD network is trained separately and its target is only to isolate different faults, thus the tracking performance of the system is not considered, and the network may exceed the output limit of the system before running 100 steps. This issue will be addressed later in the fitting network.

Figure 5.

The test result of AFD network (with faule model 1): (a) AFD State. (b) AFD MMD.

When training the fitting network, a fault occurs at step 0, and then MMD calculation is performed at the 50th and the 100th steps. The fault diagnosis is completed at step 100, leaving 400 steps to test the controlling performance of the AFDT network. Figure 6 shows the learning curve during the training process (The training of the AFDT network takes 3000 trajectories), and it can be seen that the final total reward converges to a stable value, indicating that the policy has converged. Figure 7 shows the ability of the AFDT network for fault diagnosis and control performance, in Figure 7(a), the MMD is calculated at step 50 and fault system 2 is excluded. Similarly, fault system 1 is excluded at step 100, the AFDT network identified the fault type as model 3. Besides, the tracking error is presented in Figure 7(b), it is obvious that the tracking error is kept within 0.1 rad in 500 steps, maintaining the controlling ability.

Figure 6.

The learning curve of AFDT network: (a) AFDT reward. (b) AFDT step.

Figure 7.

The result of AFDT(With fault model 3): (a) AFDT MMD. (b) AFDT track error.

To illustrate the diagnostic capability of this network for the remaining types of faults, we tested the other two types of faults with random initial condition, which is presented in Figure 8. At the 100th step, the fault model was identified. It must be acknowledged that diagnosing faults while maintaining control performance under fault model 1 presents significant challenges. This difficulty arises because the controller’s performance is severely limited, impeding its ability to achieve both diagnostic and control objectives concurrently. Nevertheless, it is noteworthy that the tracking error remains stabilized within a range of 0.1 rad, indicating that some degree of control performance is still maintained. We believe that the AFDT network can effectively identify fault types and ensure certain controlling performance.

Figure 8.

The other result of AFDT: (a) AFDT MMD (With fault model 1), (b) AFDT track error (With fault model 1), (c) AFDT MMD (With fault model 2) and (d) AFDT track error (With fault model 2).

Finally, to highlight the advantages of the AFDT method, a comparative experiment was conducted against the methodology outlined by Yan et al. (2023). The initial states for both methods were limited within the range of [–0.05, 0.05] (rad), with a simulation duration of 500 steps, each lasting 0.005 s. The evaluation refers to examining fault diagnosis and tracking performance under three fault models. The simulation results are depicted in Figures 9 and 10.

Figure 9.

The fault diagnosis performance of AFDT and the compared method: (a) AFDT MMD (With fault model 1), (b) Compared method MMD (With fault model 1), (c) AFDT MMD (With fault model 2), (d) Compared method MMD (With fault model 2), (e) AFDT MMD (With fault model 3) and (f) Compared method MMD (With fault model 3).

Figure 10.

The tracking performance of AFDT and the compared method: (a) Track error of AFDT and the compared method (With fault model 1), (b) Track error of AFDT and the compared method (With fault model 2) and (c) Track error of AFDT and the compared method (With fault model 3).

Figure 9 illustrates that both methods successfully complete fault diagnosis within 100 steps without error diagnosis. Generally speaking, the only difference between these two methods is the fluctuation of MMD value, which does not impact the diagnosis outcome. Figure 9(b) is terminated early due to the vertical angle surpassing the predefined threshold.

In Figure 10, it is evident that AFDT exhibits better tracking performance. When compared to the methodology by Yan et al. (2023), the AFDT approach demonstrates reduced error fluctuations under fault models 2 and 3. Despite severe actuator constraints in fault mode 1, the AFDT method maintains stability, whereas the comparative method loses control after 175 steps.

The primary strength of AFDT lies in its ability to flexibly adjust the weight between tracking and fault diagnosis performance. In this simulation case, the fault models cover a wide range of actuator faults, especially, fault model 1 represents a great control signal attenuation fault. When compared to the approach by Yan et al. (2023), AFDT demonstrates superior tracking performance while maintaining a similar fault diagnosis performance.

Conclusion and future work

In this article, a data-driven approach for auxiliary signal design in AFD of nonlinear systems, ensuring guaranteed control performance is proposed. In order to facilitate the design, two AC networks were established to achieve fault diagnosis and controlling targets respectively. Then, in order to balance the contradiction between the two optimization indicators, a deep reinforcement learning method is proposed to generate an integrated output. With this output, it can realize the fault diagnosis of the system while ensuring certain controlling performance. Finally, a simulation of the Cart-Pole system was conducted to verify the effectiveness of the AFDT network.

It is important to note that the proposed method requires a reference control signal to ensure effective control performance, which may be challenging to implement for certain systems. In addition, the current approach is constrained by a limited number of fault types. In future work, we aim to extend the method to accommodate a broader range of fault types and to explore strategies for maintaining satisfactory control performance without relying on a reference signal.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (grant no. 62273116)

ORCID iDs

Siyang Zhao

Jinyong Yu

Data availability statement

Data sharing not applicable to this article as no data sets were generated or analyzed during this study.

References

Andjelkovic

Campbell

(2011) Direct optimization determination of auxiliary test signals for linear problems with model uncertainty. In: IEEE conference on decision and control, Orlando, FL, 12–15 December, pp. 909–914. New York: IEEE.

Barto

Sutton

Anderson

(1983) Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems Man & Cybernetics 13(5): 834–846.

Blanke

Kinnaert

Lunze

, et al. (2006) Diagnosis and Fault-Tolerant Control. Berlin: Springer.

Chiang

Russell

Braatz

(2001) Fault Detection and Diagnosis in Industrial Systems. London: Springer.

Guo

(2024) Active diagnosis of incipient actuator faults for stochastic systems. IEEE Transactions on Industrial Electronics 71(1): 996–1005.

Heirung

TAN

Mesbah

(2019) Input design for active fault diagnosis. Annual Reviews in Control 47: 35–50.

Hellman

Raviv

(1970) Probability of error, equivocation, and the Chernoff bound. IEEE Transactions on Information Theory 16(4): 368–372.

Jia

Cao

Lyu

, et al. (2023) A novel framework of cooperative design: Bringing active fault diagnosis into fault-tolerant control. IEEE Transactions on Cybernetics 53(5): 3301–3310.

Lee

Liang

Wang

(2022) Optimal online liveness fault detection for multilayer cloud computing systems. IEEE Transactions on Dependable and Secure Computing 19(5): 3464–3477.

10.

Cao

Chen

, et al. (2023) Graph neural network architecture search for rotating machinery fault diagnosis based on reinforcement learning. Mechanical System and Signal Processing 202: 1107001.

11.

Liu

Reviriego

, et al. (2020) Scheme for periodical concurrent fault detection in parallel CRC circuits. IET Computers & Digital Techniques 14(2): 80–85.

12.

Liu

Gao

, et al. (2024) B-spline wavelet neural-network-based adaptive control for linear-motor-driven systems via a novel gradient descent algorithm. IEEE Transactions on Industrial Electronics 71(2): 1896–1905.

13.

Nikoukhah

(1998) Guaranteed active failure detection and isolation for linear dynamical systems. Automatica 34(11): 1345–1358.

14.

Schulman

Levine

Moritz

, et al. (2015) Trust region policy optimization. Computer Science 1: 1889–1897.

15.

Schulman

Wolski

Dhariwal

, et al. (2017) Proximal policy optimization algorithms. arXiv. Epub ahead of print 28 August. DOI: 10.48550/arXiv.1707.06347.

16.

Schölkopf

Platt

Hofmann

(2007) A Kernel Method for the Two-Sample-Problem. (Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference). Cambridge, MA: MIT Press.

17.

Scott

Campbell

(2014) Auxiliary signal design for failure detection in high index differential-algebraic equations. Numerical Algebra, Control & Optimization 4(2): 151–179.

18.

Silver

Lever

Heess

, et al. (2014) Deterministic policy gradient algorithms. International Conference on Machine Learning 32: 387–395.

19.

Song

Sun

Song

, et al. (2023) Quantized neural adaptive finite-time preassigned performance control for interconnected nonlinear systems. Neural Computing & Applications 35(21): 15429–15446.

20.

Sutton

McAllester

Singh

, et al. (1999) Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 12: 1057–1063.

21.

Tabatabaeipour

(2015) Active fault detection and isolation of discrete-time linear time-varying systems: a set-membership approach. International Journal of Systems Science 46(11): 1917–1933.

22.

Taqvi

SAA

Zabiri

Tufa

, et al. (2021) A review on data-driven learning approaches for fault detection and diagnosis in chemical processes. ChemBioEng Reviews 8(3): 239–259.

23.

Tawarmalani

Sahinidis

(2002) Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming, vol. 65. London: Springer.

24.

Van Hasselt

Guez

Silver

(2016) Deep reinforcement learning with double Q-learning. Thirtieth AAAI Conference on Artificial Intelligence 2016: 2094–2100.

25.

Venkatasubramanian

Rengaswamy

Kavuri

(2003a) A review of process fault detection and diagnosis. Part II: Qualitative models and search strategies. Computers & Chemical Engineering: An International Journal of Computer Applications in Chemical Engineering 27(3): 313–326.

26.

Venkatasubramanian

Rengaswamy

Kavuri

, et al. (2003b) A review of process fault detection and diagnosis. Part III: Process history based methods. Computers & Chemical Engineering: An International Journal of Computer Applications in Chemical Engineering 27(3): 327–346.

27.

Venkatasubramanian

Rengaswamy

Yin

, et al. (2003c) A review of process fault detection and diagnosis. Part I: Quantitative model-based methods. Computers & Chemical Engineering: An International Journal of Computer Applications in Chemical Engineering 27(3): 293–311.

28.

Wang

Zhuang

Tao

, et al. (2023a) Q-learning based fault estimation and fault tolerant iterative learning control for MIMO systems. ISA Transactions 142: 123–135.

29.

Wang

Liang

, et al. (2024) Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems 35(4): 5064–5078.

30.

Wang

Schaul

Hessel

, et al. (2016) Dueling network architectures for deep reinforcement learning. International Conference on Machine Learning 48: 1995–2003.

31.

Wang

Zhang

Tang

, et al. (2023b) Transfer reinforcement learning method with multi-label learning for compound fault recognition. Advanced Engineering Informatics 55: 101818.

32.

Williams

(1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8: 229–256.

33.

(2021) Observer-based asymptotic active fault diagnosis: A two-layer optimization framework. Automatica 128: 109558.

34.

(2024) A new input design framework for asymptotic active fault diagnosis with application to integrated diagnosis and control. Automatica 162: 111519.

35.

Yan

Tan

, et al. (2023) Reinforcement learning-based integrated active fault diagnosis and tracking control. ISA Transactions 132: 364–376.

36.

Yan

(2019) Data-driven load frequency control for stochastic power systems: A deep reinforcement learning method with continuous action search. IEEE Transactions on Power Systems 34(2): 1653–1656.

37.

Zhang

Chadli

Xiang

(2024a) Prescribed-time adaptive fuzzy optimal control for nonlinear systems. IEEE Transactions on Fuzzy Systems 32(4): 2403–2412.

38.

Zhang

Xiang

(2025) Prescribed-time optimal control for a class of switched nonlinear systems. IEEE Transactions on Automation Science and Engineering 22: 3033–3043.

39.

Zhang

Yan

Zou

, et al. (2024b) Fuzzy optimal tracking control for autonomous surface vehicles with prescribed-time convergence analysis. IEEE Transactions on Fuzzy Systems 32(11): 6523–6533.

40.

Zhao

Wang

Zou

, et al. (2024) Zonotopic non-fragile set-membership fusion estimation for nonlinear systems under sensor resolution effects: Boundedness and monotonicity. Information Fusion 105: 102232.

41.

Zhao

Wang

Zou

, et al. (2025) Zonotope-based distributed set-membership fusion estimation for artificial neural networks under the dynamic event-triggered mechanism. IEEE Transactions on Neural Networks and Learning Systems 36(1): 1637–1650.

42.

Zhu

Zhao

Zhang

, et al. (2024) A review of statistical-based fault detection and diagnosis with probabilistic models. Symmetry-Basel 16(4): 455.