Constrained policy optimization for adaptive high-order sliding mode control

Abstract

This paper addresses the challenge of achieving stable and constraint-satisfying control in high-order nonlinear systems. While terminal sliding mode control ensures the finite-time convergence, it often suffers from chattering and input saturation in practical implementations. In order to improve this, we propose a reinforcement learning framework based on constrained policy optimization to replace the discontinuous switching term with a learned policy, preserving sliding convergence while ensuring smooth and bounded control inputs. In addition, a theoretical input bound for high-order systems is derived and incorporated into the learning process as a safety constraint. Simulations validate the approach, showing reduced control effort and improved convergence within state limits across various initial conditions. This work provides a unified framework for safe and adaptive control, with potential applications in robotics and autonomous systems.

Keywords

constrained policy optimization terminal sliding mode reinforcement learning safe control high-order systems

Introduction

Sliding mode control (SMC) is a robust nonlinear control strategy that drives system states onto a predefined manifold and maintains their motion along this surface (Utkin, 2003). Since its development, SMC has been widely applied in fields such as autonomous driving, robotic manipulation, and aerospace systems (Guo and Woo, 2003; Jafarov and Tasaltin, 2000; Lukes, 1969; Perruquetti and Barbot, 2002; Slotine and Li, 1991), due to its strong robustness against uncertainties and external disturbances (Emel’yanov et al., 1996; Plestan et al., 2008; Zheng and Tian, 2025).

Among its variants, high-order sliding mode control (HOSMC) has attracted significant attention for its improved performance and reduced chattering (Emel’yanov et al., 1996; He et al., 2018; Plestan et al., 2008; Xi and Hesketh, 2010; Zheng and Tian, 2025). However, the recursive nature of high-order sliding surfaces amplifies higher-order terms in the control law, causing control input overflow or exponential growth beyond actuator limits. Large sampling intervals can lead to oscillations, whereas smaller intervals greatly increase computational cost. Furthermore, parameter sensitivity, dependence on initial conditions, and noise amplification remain unresolved (He et al., 2023; Ríos et al., 2021; Wang et al., 2024; Zhang et al., 2020b). These issues highlight the need for new structures or adaptive mechanisms to improve the robustness and feasibility of HOSMC in discrete implementation.

One promising direction to address these challenges is reinforcement learning (RL), which can autonomously explore optimal policies without complete system knowledge, thus handling the uncertainty and nonlinearity inherent in such systems (Bian and Jiang, 2021; Luo et al., 2016; Pei et al., 2024; Rao et al., 2023). However, standard RL methods focus primarily on maximizing cumulative rewards, often without considering safety constraints, bounded states, or input limits. These factors are crucial for HOSMC to prevent instability and actuator saturation (Liu et al., 2022a; Sun et al., 2025a; Zhao et al., 2023).

To address this limitation, we propose constrained policy optimization (CPO), which explicitly incorporates safety and performance constraints into the learning process. Specifically, CPO uses mechanisms such as Lagrange multipliers to enforce limits on state magnitudes or control amplitudes during policy updates (Achiam et al., 2017; Liu et al., 2022b; Stooke et al., 2020; Zhang et al., 2020a). This enables CPO to ensure both optimal performance and adherence to safety constraints, making it particularly suitable for practical SMC applications where constraint-set or trust-region restrictions are critical for stability. Thus, it provides an effective solution to the limitations of standard RL methods in control applications.

This work integrates CPO-based RL with state magnitude and input constraints in high-order terminal SMC (TSMC), enforcing convergence constraints on all system states and enabling adaptive controller adjustment. Additionally, a refined estimate for the upper bound of the sliding mode magnitude is introduced, improving the reliability of control input limits under high-order conditions. The actor–critic (AC) framework is employed to approximate policy and value functions, enhancing adaptive and optimal control under uncertainty and nonlinearity. The standard AC structure is chosen for its analytical clarity in studying the coupling between CPO constraints and sliding-mode requirements, as well as its compatibility with advanced RL variants. Simulation results demonstrate that the proposed method ensures global convergence and suppresses control overflow, thereby improving the robustness of high-order discrete SMC (DSMC) in complex scenarios. Moreover, it achieves reduced control effort compared with traditional methods, particularly under diverse initial conditions.

The remainder of this paper is organized as follows. Section “TSMC based on Euler’s discretization” introduces the discrete TSMC (DTSMC) model. Section “CPO-based AC method design for TSMC” details the constrained AC-based control scheme. Section “Simulation results and discussion” presents the simulation results and analysis. Section “Conclusion” concludes the study.

TSMC based on Euler’s discretization

System model

Consider the following n-dimensional SISO system in canonical form (Li et al., 2013)

\begin{matrix} {\begin{matrix} ẋ_{i} = x_{i + 1}, i = 1, \dots, n - 1 \\ ẋ_{n} = - \sum_{i = 1}^{n} c_{i} x_{i} + u + d (t) \end{matrix} \end{matrix}

(1)

where $x \in ℝ^{n}$ is the state vector, u is the control input, the positive weighting coefficients $c_{i}$ are assumed to be exactly known, and $| d (t) | \leq l \in N^{+}$ is a bounded disturbance.

Assume that the vector field is locally Lipschitz and let $h > 0$ be sufficiently small, denoting the sampling interval. Applying Euler discretization with h yields the following

\begin{matrix} {\begin{matrix} x_{i} (k + 1) = x_{i} (k) + h x_{i + 1} (k), i = 1, \dots, n - 1 \\ x_{n} (k + 1) = x_{n} (k) - h \sum_{i = 1}^{n} c_{i} x_{i} (k) + hu (k) + hd (k) \end{matrix} \end{matrix}

(2)

Here, forward Euler is chosen for its explicit form and ease of implementation in digital controllers. The sampling of the continuous state $x_{i} (t)$ at uniform intervals $t_{k} = kh$ ( $k \in N_{0}$ , with sampling time $h > 0$ ) gives the discrete time sequence $d_{k} = d (t_{k})$ and $x_{i} (k) = x_{i} (t_{k})$ . Throughout the paper $ẋ_{i} (t)$ refers to the physical derivative, while the forward difference $Δ s (k) ≜ s (k + 1) - s (k)$ is used in the digital controller. Higher-order differences are defined recursively as $Δ^{r} s (k) = Δ (Δ^{r - 1} s (k))$ , where $r \in N^{+}$ denotes a positive integer order. The control input is kept constant over $[t_{k}, t_{k + 1})$ via a zero-order hold. Then, a recursive terminal sliding surface is constructed as

\begin{matrix} s_{i + 1} (k) = Δ s_{i} (k) + β_{i + 1} s_{i}^{q_{i + 1} ∕ p_{i + 1}} (k), i = 0, \dots, n - 2 \end{matrix}

(3)

where $s_{0} = x_{1}$ , $q_{i}, p_{i}$ are odd positive integers and $0 < q_{i} ∕ p_{i} < 1$ . The control law is designed as follows

\begin{matrix} \begin{matrix} u (k) = \sum_{i = 1}^{n} c_{i} x_{i} (k) - \sum_{i = 0}^{n - 2} β_{i + 1} Δ^{n - i - 1} (s_{i}^{q_{i + 1} ∕ p_{i + 1}} (k)) \\ - K sign (s_{n - 1} (k)), K > l \end{matrix} \end{matrix}

(4)

There exists an integer $T_{SMC} \in ℕ$ such that for all $k \geq T_{SMC}$ , the following bounds hold

\begin{matrix} | s_{i} (k) | \leq ρ_{i}, | x_{1} (k) | \leq ρ_{0}, | x_{i} (k) | \leq δ_{0, i - 1}, i = 2, \dots, n \end{matrix}

(5)

and

\begin{matrix} {\begin{matrix} δ_{i, 1} = ρ_{i + 1} + β_{i + 1} ρ_{i}^{q_{i + 1} ∕ p_{i + 1}}, i = 0, \dots, n - 2 \\ δ_{i, j + 1} = δ_{i + 1, j} + β_{i + 1} {(\frac{2}{h})}^{j - (q_{i + 1} ∕ p_{i + 1})} δ_{i, 1}^{q_{i + 1} ∕ p_{i + 1}} \\ j = 1, \dots, n - 2, i = 0, \dots, n - 1 - j \end{matrix} \end{matrix}

(6)

where $s_{i} (k)$ denotes the $i th$ recursive sliding variable constructed according to the terminal sliding surface definition. $ρ_{i}$ is the ultimate bound for $s_{i} (k)$ , determined by the gains of the sliding mode, the surface parameters, and the sampling period h. $x_{1} (k)$ is the primary state of the system, with the ultimate bound $ρ_{0}$ . $x_{i} (k)$ are the higher-order system states for $i = 2, \dots, n$ , whose ultimate bounds $δ_{0, i - 1}$ are calculated recursively based on the lower-order bounds and the design parameters.

All constants $ρ_{i}$ and $δ_{0, i - 1}$ are explicit functions of the system and sliding surface parameters (such as $c_{i}$ , $β_{i + 1}$ , $q_{i} ∕ p_{i}$ , and K), as well as the sampling interval h. In particular, as h decreases and the parameters are properly tuned, these limits can be arbitrarily reduced, ensuring that all the states of the system and the sliding variables ultimately remain in a compact neighborhood around the origin (Hou et al., 2019). This property guarantees the ultimate practical boundedness of the closed-loop DTSMC system, which is fundamental for stability and performance in high-order discrete control applications. In practice, however, achievable tightness is limited by hardware sampling rates, actuator saturation, and noise amplification, so one seeks the smallest implementable bounds that still guarantee robust stability and performance.

Analysis of control law behavior

In discrete HOSMC, reducing h or increasing the gains indeed improves the numerical stability, but it also magnifies the terms of high-order difference in the control law (Abidi et al., 2007; Galias and Yu, 2007; He and Bai, 2024; Janardhanan and Bandyopadhyay, 2006; Wang and Yu, 2009). From (5) and (6), we obtain the conventional limit $δ_{m - 1, n - m} : = M_{m} \propto h^{- (n - m)}$ . Therefore, as the sampling interval h decreases, the maximum admissible control effort increases rapidly, making the classical estimate $h^{- n}$ almost impractical for high-dimensional systems. In practice, the designer must therefore trade off a small h (desired for stability) against actuator limits, tuning $β_{i}, δ_{i, j}, h$ , and related parameters accordingly.

Because the control input u is the only physically constrained signal, accurately predicting its feasible range is crucial. When the order of the system n increases, maintaining stability with a small h can drive u to scale exponentially with n, leading to prohibitive actuation requirements. Hence, while high-order discrete TSMC guarantees convergence in theory, its practical deployment hinges on tighter, more realistic input bounds.

To resolve this, we derive an improved upper bound expression for u which grows much more slowly than the traditional $h^{- n}$ term, producing a meaningful limit even for large n (Li et al., 2013). Compared with (4), we identify $m = i + 1$ . The complete derivation is provided in Appendix 1.

Theorem 1. For each order $(m = 1, 2, \dots, n)$ , define

\begin{matrix} M_{m} : = \frac{2^{1 - \frac{q_{m}}{p_{m}}}}{(n - m)! h^{n - m}} [A_{m, 0} + \sum_{ℓ = 1}^{m - 1} 2^{(1 - \frac{q_{ℓ}}{p_{ℓ}}) \frac{q_{m}}{p_{m}}} β_{ℓ}^{\frac{q_{m}}{p_{m}}} A_{m, ℓ}] \end{matrix}

(7)

The direct term $A_{m, 0}$ depends only on the outermost higher-order states $x_{j + m} (0)$

\begin{matrix} A_{m, 0} : = \sum_{j = 1}^{n - m} (\begin{matrix} n - m \\ j \end{matrix}) {| h^{j} x_{j + m} (0) |}^{q_{m} ∕ p_{m}} \end{matrix}

(8)

Here $(\begin{matrix} a \\ b \end{matrix})$ denotes the binomial coefficient $(\begin{matrix} a \\ b \end{matrix}) = \frac{a!}{b! (a - b)!}$ for non-negative integers $a \geq b$ , and is taken to be zero when $b > a$ . Each cascade term $A_{m, ℓ} (ℓ = 1, \dots, m - 1)$ pulls the index one step forward and uses a composite exponent

\begin{matrix} A_{m, ℓ} : = \sum_{j = 1}^{n - m} (\begin{matrix} n - m \\ j \end{matrix}) {| h^{j} x_{j + m - 1} (0) |}^{q_{m} q_{ℓ} ∕ p_{m} p_{ℓ}} \end{matrix}

(9)

For every $m = 1, 2, \dots, n$

\begin{matrix} | Δ^{n - m} [s_{m - 1}^{q_{m} ∕ p_{m}} (0), \dots, s_{m - 1}^{q_{m} ∕ p_{m}} (n - m)] | \leq M_{m} \end{matrix}

(10)

This theoretical result provides a more accurate upper bound for the maximum control input in high-order discrete sliding-mode systems. In subsequent integration with the RL and CPO methods, this bound can serve as either a hard or soft constraint within the reward or cost function and can be directly used to set the allowed range of control input during RL training. In addition, it offers a reliable reference baseline or disturbance margin for the hybrid control strategy. This facilitates safer and more efficient policy optimization, ensuring that the learned controller operates within theoretically guaranteed safe regions. In the following, we will provide the structure of the RL process based on this work.

CPO-based AC method design for TSMC

Sliding mode system design

Based on the previously defined discrete system in (2), we adopt a modified structure as follows

\begin{matrix} {\begin{matrix} x_{i} (k + 1) = x_{i} (k) + h x_{i + 1} (k), i = 1, 2, \dots, n - 1 \\ x_{n} (k + 1) = x_{n} (k) + hf (k, x (k)) \\ + h (\sum_{i = 0}^{n - 2} β_{i + 1} Δ^{n - i - 1} (s_{i}^{q_{i + 1} ∕ p_{i + 1}} (k)) + u_{oc} (k)) \end{matrix} \end{matrix}

(11)

In the proposed control law (4), the traditional term $K sign (\cdot)$ is replaced by a learned component, generated directly by the actor network in the AC architecture and corresponding to the compensation term $u_{oc} (t)$ . The nominal system dynamics $f (k, x (k))$ is approximated as $h \sum_{i = 1}^{n} c_{i} x_{i} (k)$ , which in practice can be further simplified via model reduction or local linearization, especially under small disturbances or partially unknown dynamics. The resulting modeling errors are adaptively compensated by the RL-based control, since the AC reward function depends only on the observed system state, without explicit reliance on system models or parameters (Hwang et al., 2025). This enables the RL component to effectively handle model uncertainties and external disturbances, providing adaptive compensation based on actual system performance.

To further enhance safety and compliance with constraints, the actor network’s policy update process is enhanced using the CPO algorithm, which enables the explicit enforcement of state and input constraints during learning. With this comprehensive DTSMC framework, the overall AC-CPO control structure and network design are developed, ensuring both fast convergence and explicit constraint satisfaction for high-order sliding-mode systems.

Actor and critic network design

The actor network forms the core of the policy gradient method, parameterizing the deterministic or stochastic policy for the closed-loop system under AC control. For the standard AC framework, a wide range of well-established design and implementation methodologies already exist in the literature. Therefore, this paper does not aim to review the complete derivation of conventional AC algorithms. Instead, we present a concise design framework and the essential mathematical formulations required for the proposed control architecture. Detailed explanations are provided only where necessary to highlight the methodological innovations introduced in this work.

In the standard AC approach, the actor is trained to generate the control input by maximizing the expected cumulative reward (Haarnoja et al., 2018; Lillicrap et al., 2015). For DSMC, this traditionally involves replacing the hand-crafted switching control law, specifically the $K sign (s_{n - 1})$ term in the SMC formulation, with the output of a neural network such that

\begin{matrix} a_{k} = u_{oc} (k) = π_{θ} (x (k)) \end{matrix}

(12)

where $x (k) \in ℝ^{n}$ is the sampled state, θ denotes the parameters of the actor network, and $π_{θ}$ is the parameterized policy. The generated output $u_{oc} (k)$ is applied directly as the switching term in the control law

\begin{matrix} u (k) & = \sum_{i = 1}^{n} c_{i} x_{i} (k) - \sum_{i = 0}^{n - 2} β_{i + 1} Δ^{n - i - 1} (s_{i}^{q_{i + 1} ∕ p_{i + 1}} (k)) \\ + u_{oc} (k) \end{matrix}

(13)

The policy improvement step for the actor network is typically achieved by maximizing the expected advantage function over the current policy (Yasser et al., 2006), formulated as

\begin{matrix} \max_{π \in Π} E_{s \sim d^{π_{k}}, a \sim π} [A^{π_{k}} (s, a)] \end{matrix}

(14)

where $A^{π_{k}} (s, a)$ denotes the advantage function under the current policy $π_{k}$ and $d^{π_{k}}$ is the discounted state distribution under $π_{k}$ . Here, Π denotes the set of admissible neural policies parameterized by θ. However, this objective does not inherently guarantee the satisfaction of safety or engineering constraints, such as limits on state amplitudes or control input magnitudes, which are crucial in real-world HOSMC systems.

To address these practical requirements, the actor update is performed under explicit constraints in the CPO framework. For each engineering index i, such as state magnitude or input amplitude, the expected cumulative constraint cost is enforced to remain within a specified bound with discount factor $0 < γ < 1$

\begin{matrix} J_{C_{i}} (π_{k}) + \frac{1}{1 - γ} E_{s \sim d^{π_{k}}, a \sim π} [A_{C_{i}}^{π_{k}} (s, a)] \leq d_{i} \end{matrix}

(15)

where $J_{C_{i}} (π_{k})$ denotes the cumulative constraint cost for the $i th$ constraint, and $d_{i}$ represents the associated upper bound (Chow et al., 2019; Ding et al., 2023; Ma et al., 2024; Zhang et al., 2022). This formulation directly guarantees the safety and operational limits that are essential in engineering applications but are not inherently considered in standard AC approaches.

To further enhance the stability of policy updates and prevent abrupt parameter changes in the actor network, a trust region constraint is introduced by bounding the KL divergence between the updated and previous policies

\begin{matrix} E_{s \sim d^{π_{k}}} [D_{KL} (π (\cdot | s) ∥ π_{k} (\cdot | s))] \leq δ \end{matrix}

(16)

Here $d^{π_{k}}$ is induced by the old policy, bounding its expected KL divergence ensures that each update stays within a trust region of radius δ (Bhat and Bernstein, 2000; Mei et al., 2026b; Zhu et al., 2022). By explicitly incorporating these constraints into the CPO-based actor optimization, the resulting control policy achieves both optimal performance and strict compliance with safety and state/input magnitude requirements.

In summary, while the standard actor method focuses solely on reward maximization, the proposed CPO-based approach augments the objective with essential real-world constraints for robust and safe operation. The critic structure remains unchanged, continuing to estimate reward and constraint value functions, whereas the actor update is fundamentally redefined under CPO. This integration establishes a balance between performance optimization and constraint satisfaction, meeting the requirements of high-order discrete SMC and enabling safe and efficient learning-based control.

The most basic AC method employs a single critic network to estimate either the action value function $Q_{ϕ} (s, a)$ or the state value function $V_{ψ} (s)$ , where ϕ and ψ are the network parameters (Lillicrap et al., 2015). The corresponding loss function is the following

\begin{matrix} J_{V} (ψ) = E_{(s_{t}, a_{t}, r_{t}, s_{t + 1}) \sim D} [1 ∕ 2 {(V_{ψ} (s_{t}) - (r_{t} + γ V_{ψ} (s_{t + 1})))}^{2}] \end{matrix}

(17)

where D is the replay buffer that contains experience tuples $(s_{t}, a_{t}, r_{t}, s_{t + 1})$ , $r_{t}$ is the immediate reward, and γ is the discount factor.

Alternatively, when using the action-value function, the critic minimizes the following

\begin{matrix} \begin{matrix} J_{Q} (ϕ) & = E_{(s_{t}, a_{t}, r_{t}, s_{t + 1}) \sim D} [1 ∕ 2 (Q_{ϕ} (s_{t}, a_{t}) \\ - (r_{t} + γ Q_{ϕ} (s_{t + 1}, a_{t + 1})))^{2}] \end{matrix} \end{matrix}

(18)

The action $a_{t + 1}$ is sampled from the current policy and stored together with the transition so that the bootstrapped one-step target can be computed off-policy.

The gradients of the critic loss functions are computed and used to update the network via standard optimization methods, and the resulting value estimates are further employed to calculate policy gradients for updating the actor network. As the critic in the AC-CPO framework primarily serves for value estimation and policy evaluation, this work adopts a standard update scheme without additional architectural modifications. Specifically, a single critic network is used, without entropy regularization, double Q-networks, or target networks, ensuring theoretical clarity and isolating its role from the CPO-based policy update mechanism.

It is worth noting that the key contribution of CPO lies in modifying the actor update to enforce constraint satisfaction, while the critic structure and update rules remain unchanged. Consequently, the critic update is conceptually decoupled from constraint enforcement and can be readily extended with advanced techniques, such as target networks for improved stability, twin critics (TD3) for reducing overestimation bias, or entropy-augmented value functions (SAC). These extensions can be incorporated without affecting the fundamental CPO-based actor optimization. For conciseness, such implementations are not detailed here.

Coordinated policy optimization with constraint enforcement

The coordinated structure between the actor and critic modules is characterized by a dynamic interplay of policy evaluation and improvement, where the actor’s policy is iteratively updated based on feedback signals provided by the critic. This interaction forms the backbone of the AC-CPO framework, ensuring both performance optimization and constraint satisfaction throughout the training process.

In the critic module, the parameters ϕ are updated by minimizing the Bellman error between the predicted value and the target, based on sampled transitions from the replay buffer. The update gradient for the critic network is given by

\begin{matrix} \nabla_{ϕ} J_{Q} (ϕ) = \nabla_{ϕ} Q_{ϕ} (s_{t}, a_{t}) (Q_{ϕ} (s_{t}, a_{t}) - Q_{target} (s_{t}, a_{t})) \end{matrix}

(19)

Here, $Q_{target} (s_{t}, a_{t}) = r_{t} + γ Q_{ϕ} (s_{t + 1}, a_{t + 1})$ , with $a_{t + 1}$ sampled from the current policy.

For the actor network, which in this framework is optimized using the CPO algorithm, the update aims to maximize the expected advantage while enforcing explicit engineering and safety constraints. At each iteration, the actor network parameters θ are updated by solving the following constrained optimization problem

\begin{matrix} \max_{π \in Π} E_{s \sim d^{π_{k}}, a \sim π} [A^{π_{k}} (s, a)] \end{matrix}

(20)

This optimization is performed subject to explicit bounds on cumulative constraint costs and a trust region on policy updates. These constraints are imposed to ensure that the learned policy not only maximizes long-term rewards but also satisfies all required safety and operational specifications.

During training, the critic network is updated using sampled transitions to estimate the action value function, providing essential feedback for policy optimization. Simultaneously, the actor network is optimized via the CPO algorithm to produce the SMC switching control action, with policy updates carried out under explicit state and input constraints. This coordinated update scheme enables the policy to achieve high performance and constraint satisfaction throughout the learning process. The experience replay buffer stores tuples $(s_{t}, a_{t}, r_{t}, s_{t + 1}, a_{t + 1})$ , where actions are drawn from the current policy, improving sample efficiency and stabilizing learning by decorrelating consecutive experiences, which is essential for reliable optimization in high-dimensional DSMC.

In summary, the AC-CPO framework enables efficient and safe learning by integrating a CPO actor with a value-based critic. To assess the performance of the proposed CPO-based control method, we now turn to the simulation results. The following section presents simulations under various initial conditions, demonstrating system convergence, suppression of control overflow, satisfaction of safety constraints, and comparison with traditional methods.

Simulation results and discussion

Simulation model description

In practical applications of SMC, second-order systems are the most common, while higher-order systems ( $n \geq 3$ ) are significantly more difficult to control due to challenges in ensuring global convergence and stability. These difficulties stem from rapidly increasing control input magnitudes, heightened sensitivity to parameter selection, and amplified modeling or discretization errors. Moreover, higher-order systems require greater computational resources and are more prone to numerical instability, limiting their feasibility in real-time implementations. Therefore, this work focuses on a third-order discrete-time system, which captures the key challenges of high-order control while remaining tractable for analysis and simulation. For higher-order models, the primary impact is increased computational burden, whereas the fundamental convergence mechanism remains unchanged.

Although RL is well suited for compensating unknown system components or bounded disturbances within a certain range, the simulation system in this paper is deliberately kept fully known and deterministic. This is because even for a fully known third-order system, ensuring reliable convergence using HOSMC itself is a highly nontrivial challenge. Furthermore, RL does not rely on prior knowledge of system dynamics. That is, the existence or absence of explicit disturbance terms does not affect the ability of RL to adapt and learn effective compensation in the present framework.

Thus, the third-order DTSMC system used in our experiments is described as follows

\begin{matrix} {\begin{matrix} x_{1} (k + 1) & = x_{1} (k) + h x_{2} (k) \\ x_{2} (k + 1) & = x_{2} (k) + h x_{3} (k) \\ x_{3} (k + 1) & = x_{3} (k) + h [\sum_{i = 0}^{1} β_{i + 1} Δ^{2 - i} (s_{i}^{q_{i + 1} ∕ p_{i + 1}} (k)) + u_{oc} (k)] \end{matrix} \end{matrix}

(21)

Here, $u_{oc} (k)$ is the compensation generated by the RL algorithm. The sampling interval h and the gains in the sliding mode $β_{i}$ are chosen to ensure numerical stability, following established guidelines in the control literature (e.g. $β_{i} h < 2$ , see Xi and Hesketh (2010)).

Implementation of CPO-based AC method

The numerical experiments are implemented in Python using the PyTorch library. The actor and critic networks adopt identical three-layer fully connected architectures, with each hidden layer containing 64 ReLU units. For CPO, an additional constraint-critic network of the same architecture is introduced. The Adam optimizer is used for all networks with a learning rate of $1 0^{- 3}$ , and the training is conducted over 80 episodes, each lasting 5 seconds with a step size $h = 0.01$ . The RL output is passed through a $\tanh$ function to ensure bounded actions and then scaled to meet the system input requirements.

The system environment employs coefficients $c_{1} = 3.6$ , $c_{2} = 3.0$ , $c_{3} = 1.0$ , $β_{1} = 5.0$ , $β_{2} = 3.0$ , $q_{1} ∕ p_{1} = 1 ∕ 2$ , $q_{2} ∕ p_{2} = 3 ∕ 5$ , and an RL action scale of $0.06$ . The output of the RL policy is subject to additional amplitude constraints for safety and stability. These values are selected through a combination of design principles and empirical tuning, ensuring that the control task remains sufficiently challenging for a comprehensive evaluation.

The reward function is carefully designed to balance the goals of rapid state convergence and minimizing control effort. Specifically, a quadratic penalty is assigned on each state variable, with heavier weights on $x_{1}$ to emphasize primary convergence, and a small penalty on the control input amplitude. Formally, the reward at each step is given by

\begin{matrix} r_{t} = - (220 x_{1}^{2} + 14 x_{2}^{2} + 2 x_{3}^{2} + 0.025 u^{2}) \end{matrix}

(22)

This formulation penalizes large deviations of $x_{1}$ , moderate deviations of $x_{2}$ and $x_{3}$ , and limits excessive the effort to control. A more stringent constraint cost is defined as the sum of squared state norm excesses above a set threshold, which is enforced within the CPO training loop.

It is worth mentioning that the Lagrange multiplier λ, introduced as an auxiliary variable in the constrained optimization problem, measures the impact of constraints on the objective function (Schulman et al., 2017). In the RL-CPO framework, the Lagrange multiplier is used to control the penalty for constraint costs. When constraints are violated, the multiplier increases, forcing more conservative policy updates to ensure compliance with the maintaining reward optimization constraints. Simulation outputs, including state trajectories and control inputs, are stored for post-processing and visualization.

Algorithm 1. CPO–AC-DTSMC training loop.
Require: step size h, episode count $N_{epi}$ , steps per episode T
Require: actor θ, critic ϕ (optional V–critic ψ)
Require: replay buffer $D$ , KL radius δ, cost limit d
1: initialize network weights, Lagrange multiplier $λ \leftarrow λ_{0}$
2: for episode $e \leftarrow 1$ to $N_{epi}$ do
3: reset plant state $x \leftarrow x_{0}$ ; set $C_{ep} \leftarrow 0$
4: for step $k \leftarrow 0$ to $T - 1$ do ⊳ interaction phase
5: compute sliding terms $s_{0}, s_{1}$ via forward difference
6: $u_{SMC} \leftarrow nominalSMC (s_{0}, s_{1}, β_{1}, β_{2})$
7: $u_{o c} \leftarrow \tanh (π_{θ} (x)) \times scale$
8: $u \leftarrow saturate (u_{SMC} + u_{o c})$
9: $x^{'} \leftarrow plantStep (x, u, h)$
10: $r \leftarrow reward (x, u)$
11: $c \leftarrow constraintCost (x)$ ; $C_{ep} + = c$
12: $D . append (x, u_{o c}, r, c, x^{'}, a_{t + 1})$ ; $x \leftarrow x^{'}$ ⊳ online critic update
13: sample mini-batch $B \subset D$
14: update Q-critic ϕ with TD target
15: if V-head active then
16: update state-value head ψ
17: end if
18: end for ⊳ episode-level CPO actor update
19: compute advantages $A (\cdot)$ with latest critic
20: obtain $Δ θ$ from constrained TRPO/CPO step s.t. $KL \leq δ, C_{ep} \leq d$
21: $θ \leftarrow θ + lineSearch (Δ θ)$
22: $λ \leftarrow \max (0, λ + η (C_{ep} - d))$
23: end for

To ensure complete reproducibility, the remaining training constants are fixed as follows: the discount factor is $γ = 0.97$ , the CPO trust region radius is $δ = 0.02$ , the episode-level constraint limit is $d = 0.007$ , the replay buffer stores $2 \times 1 0^{4}$ transitions, from which mini-batches of $128$ samples are uniformly drawn for each critic update, and the Lagrange multiplier is adjusted with a dual-gradient step size $η = 0.03$ . The threshold is fixed at $ξ = 0.13$ throughout this experiment. All values were selected through a coarse grid search to balance learning speed, constraint satisfaction, and numerical robustness, and are kept unchanged across all reported experiments.

For each episode, the agent first collects transitions, while the Q-critic is updated online. Afterward, a constrained trust-region step is executed to update the actor parameters. The algorithmic skeleton is implementation-agnostic and can be reproduced in any modern RL framework. If the state-value head $V_{ψ}$ is not used, the corresponding block can be removed. All symbols and coefficients are defined in the preceding sections. The effectiveness of the above procedure is evaluated in the next section via a series of Monte Carlo simulations.

All symbols and coefficients are defined in the previous sections or within the algorithm description. The auxiliary functions saturate(·), lineSearch(·), and scale follow standard engineering conventions and are not further elaborated here, representing actuator saturation, trust-region backtracking, and policy output scaling, respectively (Ding et al., 2025; Haripriya et al., 2025; Mei et al., 2026a; Schulman et al., 2015; Sun et al., 2025b, 2026; Yang et al., 2024).

Results and discussion

To examine the baseline behavior, a third-order DTSMC system is simulated using the classical control law (4), without any RL component. As shown in Figure 1, the state trajectories $x_{1}$ and $x_{2}$ exhibit bounded convergence, while $x_{3}$ undergoes sustained oscillations and divergence, indicating a breakdown in stability for higher-order components. This instability results mainly from the amplification of higher-order difference terms such as $Δ^{r} (s^{q ∕ p})$ in the control law. When $r \geq 2$ , small numerical fluctuations or discretization errors can grow rapidly, especially at limited sampling rates. Moreover, the fixed switching gain K must be set conservatively high to ensure robustness, but this often leads to excessive chattering and actuator saturation, further destabilizing higher-order states.

Figure 1.

State trajectories under traditional DTSMC without RL compensation.

These observations underscore the numerical sensitivity and practical limitations of high-dimensional DTSMC operating without adaptive mechanisms. They motivate the integration of RL-based compensation to suppress divergence and stabilize the switching dynamics.

To comprehensively evaluate the behavior of the proposed CPO-enhanced DTSMC controller, we analyze its performance under two different initial condition ranges in Figure 2(a) and (b): a small neighborhood where $| x_{i} (0) | \leq 0.02$ and a large neighborhood where $| x_{i} (0) | \leq 0.3$ . In the small initial range, all the states of the system $x_{1}$ , $x_{2}$ , and $x_{3}$ converge smoothly to the origin, demonstrating that the compensation term generated by RL $u_{oc} (k)$ effectively drives the system to the sliding manifold and achieves precise stabilization. In contrast, in the large initial range, although $x_{2}$ and $x_{3}$ still converge to zero, $x_{1}$ stabilizes at a nonzero offset. This indicates that the current parameter configuration does not fully guarantee convergence in the wider initial domain, and further parameter tuning is required.

Figure 2.

State trajectories under different initial conditions for the proposed CPO-enhanced DTSMC controller. (a) Small initial region. (b) Large initial region.

Figure 3(a) and (b) illustrates the absolute values of the states of the system along with their constraint boundaries under different initial conditions. In the small-range case, the initial condition was set to $| x_{i} (0) | = 0.2$ , which, although relatively close to the constraint threshold, still allows the three states $x_{1}$ , $x_{2}$ , and $x_{3}$ to smoothly converge to the origin. Compared with an even smaller magnitude such as $0.02$ , this setting exhibits the same qualitative convergence trend. In contrast, when the initial range increases to $| x_{i} (0) | = 0.3$ , as shown in Figure 3(b), a different behavior is observed. Although $x_{2}$ and $x_{3}$ still decay to zero, the state $x_{1}$ does not converge completely and stabilizes at a nonzero offset. This partial convergence violates the desired control objective of full state stabilization and indicates that current control parameters are no longer sufficient to handle such a high initial energy in $x_{1}$ . In this scenario, parameter re-tuning or policy adaptation becomes necessary to restore convergence capability under larger disturbances.

Figure 3.

State absolute values and constraint boundary under different initial conditions for the proposed CPO-enhanced DTSMC controller. (a) Small initial region. (b) Large initial region.

The constraint cost profiles in Figure 4(a) and (b) show that the average constraint cost remains well below the predefined limit under small initial conditions. However, in the large-range case, the average constraint cost (Avg cost/ep) gradually exceeds the red threshold line (limit) as training progresses. This violation does not necessarily mean that $x_{1}$ directly crosses the state constraint boundary, but rather that the cumulative constraint penalty increases due to sustained near-boundary behavior. As a result, the CPO algorithm tightens policy updates to enforce constraint satisfaction, which may reduce control effectiveness, especially in regulating $x_{1}$ . This indirect effect prevents $x_{1}$ from fully converging to zero, indicating that the safety mechanism limits convergence under high initial energy without further parameter tuning.

Figure 4.

Constraint cost during training under different initial conditions. (a) Small initial region. (b) Large initial region.

Finally, the learning curves in Figure 5(a) and 5(b) further support these conclusions. In the small-range case, the return improves steadily, and the Lagrange multiplier remains close to unity, indicating minimal constraint activation. In contrast, in the large-range setting, the return fluctuates and exhibits slower overall improvement, while the Lagrange multiplier grows larger over time. This behavior reflects more frequent violations of the constraint cost limit, which triggers stronger penalties and restricts policy updates.

Figure 5.

Learning curve and Lagrange multiplier evolution. (a) Small initial region. (b) Large initial region.

The Lagrange multiplier λ in the CPO framework plays a critical role in the balance of reward optimization and constraint satisfaction. A higher value indicates that the constraint cost has exceeded the threshold, and the algorithm enforces more conservative updates to ensure safety. This mechanism, while effective for preserving feasibility, also reduces the learning rate and can limit performance in extreme conditions.

As shown in Figure 6, the control input amplitude is significantly higher in the case of a large initial region compared to the small one. The input signal for the large region not only reaches higher peaks, but also exhibits sustained oscillations, suggesting that the controller operates closer to the actuator saturation limits. In contrast, under small initial conditions, the control input remains well within the allowable range, leading to smoother and more stable convergence behavior. This comparison further confirms the trade-off between input feasibility and state convergence in high-energy scenarios, highlighting the importance of appropriate constraint-aware control synthesis.

Figure 6.

Comparison of control input amplitude in small and large regions.

It is important to note that the theoretical input bound $M_{1} \approx 6.67 \times 1 0^{- 5}$ in this simulation based on (1) only reflects the magnitude of the homogeneous nonlinear term before including the switching component. In classical SMC, a large fixed gain K is typically applied to the discontinuous $sign (s)$ term, often resulting in control inputs that exceed practical actuator limits. However, in the proposed CPO-RL framework, the RL policy replaces this switching term, as evidenced in Figure 6. The resulting control input remains bounded and smooth, even under large initial deviations, demonstrating that the RL component preserves convergence while significantly reducing control effort by implicitly regulating switching behavior.

Although $M_{1}$ appears negligible in the third-order case, its value increases rapidly with system order due to its factorial and inverse power dependence on the sampling step size. In higher order systems ( $n \geq 4$ ), this bound can dominate the input dynamics, making it a critical factor in evaluating the feasibility of control under limited actuator capability. Nevertheless, even in the present third-order case, the estimated upper bound is still significantly reduced compared to the conventional method, yielding a value on the order of $1 0^{- 2}$ under the simulation conditions.

In summary, the proposed controller performs well in small-to-moderate initial ranges. However, for larger initial states, policy tuning and adaptation may be required to maintain full convergence and stability.

Conclusion

This work proposed a CPO-based RL framework to enhance DTSMC for nonlinear systems. By replacing the discontinuous switching term with a learned policy, the method ensured smooth, bounded control while preserving convergence. Theoretical input bounds for higher-order systems were also derived and validated in simulations. Within moderate initial states, the approach achieved robust convergence and reduced control effort. For larger deviations, early input saturation limits convergence, suggesting the need for adaptive parameter tuning. The method is promising for constrained applications such as robotics and unmanned aerial vehicles (UAVs). Future research will focus on extending the framework to higher-order systems, real-time adaptation, and continuous-time formulations.

Footnotes

Appendix 1 ORCID iDs

Shihong Ding

Keqi Mei

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the National Natural Science Foundation of China under grant 62203188 and grant 62373170, and in part by the National Key Research and Development Program of China under grant 2022YFD200150203.

Declaration of conflicting interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability statement

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

Abidi

(2007) On the discrete-time integral sliding-mode control. IEEE Transactions on Automatic Control 52(4): 709–715.

Achiam

Held

Tamar

, et al. (2017) Constrained policy optimization. In: Proceedings of the 34th international conference on machine learning, Sydney, NSW, Australia, 6–11 August, pp.22–31.

Bhat

Bernstein

(2000) Finite-time stability of continuous autonomous systems. SIAM Journal on Control and Optimization 38(3): 751–766.

Bian

Jiang

(2021) Reinforcement learning and adaptive optimal control for continuous-time nonlinear systems: A value iteration approach. IEEE Transactions on Neural Networks and Learning Systems 33(7): 2781–2790.

Chow

Nachum

Faust

, et al. (2019) Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 .

Ding

Wei

, et al. (2023) Disturbance-observer-based barrier function adaptive sliding mode control for path tracking of autonomous agricultural vehicles with matched-mismatched disturbances. IEEE Transactions on Transportation Electrification 10(3): 6748–6760.

Ding

Dou

Liu

, et al. (2025) Event-triggered adaptive super-twisting-like fractional control for path-tracking of autonomous agricultural vehicles. IEEE/ASME Transactions on Mechatronics 31: 1346–1356.

Emel’yanov

Korovin

Levant

(1996) High-order sliding modes in control systems. Computational Mathematics and Modeling 7(3): 294–318.

Galias

(2007) Euler’s discretization of single input sliding-mode control systems. IEEE Transactions on Automatic Control 52(9): 1726–1730.

10.

Guo

Woo

(2003) An adaptive fuzzy sliding mode controller for robotic manipulators. IEEE Transactions on Systems, Man, and Cybernetics: Systems and Humans 33(2): 149–159.

11.

Haarnoja

Zhou

Abbeel

, et al. (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th international conference on machine learning, pp.1861–1870.

12.

Haripriya

Manivannan

Dhanasekar

, et al. (2025) Finite-time synchronization of delayed complex dynamical networks via sampled-data controller. Mathematical Modelling and Control 5(1): 73–84.

13.

Lyu

Liu

(2018) Robust H sliding mode controller design of a class of time-delayed discrete conic-type nonlinear systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(2): 885–892.

14.

Bai

(2024) Finite-time stability and applications of positive switched linear delayed impulsive systems. Mathematical Modelling and Control 4(2): 178–194.

15.

(2023) Nonlinear extended state observer-based adaptive higher-order sliding mode control for parallel antenna platform with input saturation. Nonlinear Dynamics 111(17): 16111–16132.

16.

Hou

, et al. (2019) Discrete-time terminal sliding-mode tracking control with alleviated chattering. IEEE/ASME Transactions on Mechatronics 24(4): 1808–1817.

17.

Hwang

Kim

Seok

(2025) Generative adversarial soft actor–critic. IEEE Transactions on Neural Networks and Learning Systems 36: 11917–11927.

18.

Jafarov

Tasaltin

(2000) Robust sliding-mode control for the uncertain MIMO aircraft model F-18. IEEE Transactions on Aerospace and Electronic Systems 36(4): 1127–1141.

19.

Janardhanan

Bandyopadhyay

(2006) On discretization of continuous-time terminal sliding mode. IEEE Transactions on Automatic Control 51(9): 1532–1536.

20.

(2013) Discrete-time terminal sliding mode control systems based on Euler’s discretization. IEEE Transactions on Automatic Control 59(2): 546–552.

21.

Lillicrap

Hunt

Pritzel

, et al. (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

22.

Liu

Zhang

Tateo

, et al. (2022a) Safe reinforcement learning of dynamic high-dimensional robotic tasks: Navigation, manipulation, interaction. arXiv preprint arXiv:2209.13308 .

23.

Liu

Cen

Isenbaev

, et al. (2022b) Constrained variational policy optimization for safe reinforcement learning. In: Proceedings of the 39th international conference on machine learning, Baltimore, MD, 17–23 July, pp.13644–13668.

24.

Lukes

(1969) Optimal regulation of nonlinear dynamical systems. SIAM Journal on Control 7(1): 75–100.

25.

Luo

Liu

Huang

, et al. (2016) Model-free optimal tracking control via critic-only Q-learning. IEEE Transactions on Neural Networks and Learning Systems 27(10): 2134–2144.

26.

Gao

(2024) Adversarial constrained policy optimization: Improving constrained reinforcement learning by adapting budgets. arXiv preprint arXiv:2410.20786 .

27.

Mei

, et al. (2026a) Path tracking control of autonomous agricultural tractor: A non-smooth sampled-data approach. IEEE Transactions on Industrial Electronics. Epub ahead of print 5 May. https://doi.org/10.1109/TIE.2026.3654806

28.

Mei

, et al. (2026b) Prescribed-time SOSM based path-following control for autonomous land vehicles with guaranteed performance. IEEE Transactions on Vehicular Technology. Epub ahead of print 30 January. https://doi.org/10.1109/TVT.2026.3659729

29.

Pei

Zhao

, et al. (2024) Visibility-enhanced model-free deep reinforcement learning algorithm for voltage control in realistic distribution systems using smart inverters. Applied Energy 372: 123758.

30.

Perruquetti

Barbot

(2002) Sliding Mode Control in Engineering. Marcel Dekker.

31.

Plestan

Glumineau

Laghrouche

(2008) A new algorithm for high-order sliding mode control. International Journal of Robust and Nonlinear Control 18(4–5): 441–453.

32.

Rao

Wang

, et al. (2023) Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces. Nonlinear Dynamics 111(21): 20041–20053.

33.

Ríos

Franco

de Loza

, et al. (2021) A high-order sliding-mode adaptive observer for uncertain nonlinear systems. IEEE Transactions on Automatic Control 68(1): 408–415.

34.

Schulman

Levine

Abbeel

, et al. (2015) Trust region policy optimization. In: Proceedings of the 32nd international conference on machine learning, Lille, France, 6–11 July, pp.1889–1897.

35.

Schulman

Wolski

Dhariwal

, et al. (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

36.

Slotine

JJE

(1991) Applied Nonlinear Control. Prentice Hall.

37.

Stooke

Achiam

Abbeel

(2020) Responsive safety in reinforcement learning by PID Lagrangian methods. In: Proceedings of the 37th international conference on machine learning, Virtual event, 13–18 July, pp.9133–9143.

38.

Sun

Jiang

Zhang

, et al. (2025a) Multi-robot hierarchical safe reinforcement learning autonomous decision-making strategy based on uniformly ultimate boundedness constraints. Scientific Reports 15(1): 5990.

39.

Sun

, et al. (2025b) An efficient trajectory planning approach for tractor-trailer vehicles using convex polygonal corridor construction in narrow environments. IEEE Transactions on Vehicular Technology 75(1): 419–432.

40.

Sun

Yan

Ding

, et al. (2026) Adaptive second-order sliding mode control design with autonomous agricultural vehicle application. IEEE Transactions on Industrial Electronics 73: 3093–3103.

41.

Utkin

(2003) Variable structure systems with sliding modes. IEEE Transactions on Automatic Control 22(2): 212–222.

42.

Wang

(2009) Euler discretization of second-order terminal sliding mode control systems. In: Proceedings of the 48th IEEE conference on decision and control, Shanghai, China, 16–18 December, pp.6697–6702.

43.

Wang

Duan

(2024) Event-triggered adaptive sliding mode control of uncertain nonlinear systems based on fully actuated system approach. IEEE Transactions on Circuits and Systems II: Express Briefs 71(5): 2749–2753.

44.

Hesketh

(2010) On discrete time terminal sliding mode control for nonlinear systems with uncertainty. In: Proceedings of the 2010 American control conference, Baltimore, MD, 30 June–2 July, pp.980–984.

45.

Yang

Ding

(2024) Fast supertwisting sliding mode control with antipeaking extended state observer for path-tracking of unmanned agricultural vehicles. IEEE Transactions on Industrial Electronics 71(10): 12973–12982.

46.

Yasser

Trisanto

, et al. (2006) Adaptive sliding mode control using simple adaptive control for SISO nonlinear systems. In: Proceedings of the 2006 IEEE international symposium on circuits and systems, Kos, Greece, 21–24 May, p.4.

47.

Zhang

Shen

Yang

, et al. (2022) Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205.11814.

48.

Zhang

Vuong

Ross

(2020a) First order constrained optimization in policy space. Advances in Neural Information Processing Systems 33: 15338–15349.

49.

Zhang

Yang

Wei

, et al. (2020b) Fractional-order adaptive non-singular fast terminal sliding mode control with time delay estimation for robotic manipulators. IET Control Theory & Applications 14(17): 2556–2565.

50.

Zhao

Chen

, et al. (2023) State-wise safe reinforcement learning: A survey. arXiv preprint arXiv:2302.03122 .

51.

Zheng

Tian

(2025) Exponential stability of time-delay systems with highly nonlinear impulses involving delays. Mathematical Modelling and Control 5(1): 103–120.

52.

Zhu

Deng

Zheng

, et al. (2022) Online reinforcement-learning-based adaptive terminal sliding mode control for disturbed bicycle robots on a curved pavement. Electronics 11(21): 3495.