Data-driven optimal trajectory tracking control of autonomous surface vessels via reinforcement learning

Abstract

This paper presents an optimal trajectory tracking control algorithm for autonomous surface vessels (ASVs) using data-driven reinforcement learning (RL) to address challenges arising from model uncertainties and time-varying external disturbances in complex marine environments. To ensure robust performance under these conditions, we first employ the H_∞ control method. Then, we design a model-based RL algorithm to achieve the optimal trajectory tracking control law for the ASV despite uncertainties and disturbances. In addition, we extend the model-based RL algorithm to a model-free data-driven RL algorithm, removing the requirement for model of the ASV. The model-free algorithm directly learns the optimal control law from real-time data, providing a more flexible solution when the model of the ASV is unknown. Simulations are conducted to verify the proposed algorithms.

Keywords

Reinforcement learning (RL)data-driven optimal control autonomous surface vessel (ASV)trajectory tracking

Introduction

Autonomous surface vessels (ASVs), as prominent representatives of autonomous maritime platforms, possess immense potential for various applications (Hou et al., 2024; Jiang et al., 2025; Qiao et al., 2023; Yan et al., 2025). These applications include search and rescue operations, marine resource exploration and security patrols (Cao et al., 2024, 2022; Wang et al., 2024a). In these domains, the ability of ASVs to track a desired trajectory is a prerequisite for achieving mission objectives (Meng et al., 2024; Mu et al., 2024; Wang et al., 2022a,b; Zhang and Chen, 2022). Due to the significant energy demands in marine transportation and exploration, it is crucial to incorporate optimal control into ASV design. Optimal control enhances the system performance of ASVs and addresses trajectory tracking control problems (Chen et al., 2024; Lewis et al., 2012). Therefore, ensuring safe, accurate, and energy-efficient optimal trajectory tracking of ASVs has emerged as a significant challenge in marine engineering and control. Although optimal trajectory tracking for ASVs is crucial, deriving the optimal control law based on classical optimal control theory becomes exceedingly challenging because of the challenges posed by the marine environment, the inherent nonlinear characteristics of the ASV system and the uncertainties of system parameters.

The development of robust control techniques aims to address the challenges posed by uncertainties in dynamic models, ensuring that systems can maintain reliable performance despite modeling uncertainties and disturbances (Khalaji and Bahrami, 2022; Li et al., 2024; Liu and Yao, 2016; Wang et al., 2021). As a robust control method, H_∞ control aims to design control strategies capable of handling system model uncertainties and disturbances (Chen, 2013). As a result, H_∞ control has been widely applied to ASV systems (Huang et al., 2020; Rigatos, 2023). For the ASV system with parametric uncertainties and external disturbances, its course control system can be modeled as a polynomial system with polytopic uncertainties. Stability and H_∞ conditions are derived using parameter-dependent Lyapunov function methods and positive polynomial theory, enabling effective control of the ASV system (Huang et al., 2020). A nonlinear optimal control law based on H_∞ control is proposed to address the ASV control problem under model uncertainties and disturbances, aiming to enhance autonomy and reduce energy consumption (Rigatos, 2023). However, these control strategies are based on nominal ASV models and may fail to achieve optimal performance in uncertain real-world systems. The controller proposed in this paper effectively addresses uncertain systems in real-world ASV applications, achieving optimal trajectory tracking.

Classical optimal controllers typically rely on precise dynamic system models, which present challenges in their application to complex environments, particularly when the system model is difficult to obtain accurately (Lewis et al., 2012; Zhou et al., 2023a). Several approaches have been introduced to improve the trajectory tracking performance of ASVs in order to overcome these obstacles (Gu et al., 2023; Huang et al., 2024; Zhou et al., 2023b). For the optimal path tracking problem of ASVs under state constraints, a control method combining backstepping, adaptive dynamic programming and event-triggered mechanisms has been proposed by Zhou et al. (2023b). This method significantly reduces communication and computational burdens by introducing guidance laws, dynamic controllers, and event-triggered modules, ensuring near-optimal performance in the process. A technique is introduced for optimal trajectory control of ASVs with obstacles, using a fixed-time extended state observer and a recursive neural network (NN) optimization algorithm to ensure safety and achieve the desired objectives (Gu et al., 2023). Furthermore, an integrated optimal control framework is proposed to tackle the distributed optimal coordination challenge for ASVs, combining trajectory optimization for coordination with local tracking subsystems to ensure global optimality (Huang et al., 2024). However, all of these methods depend on precise dynamic system data of the ASV, which is often challenging to obtain in the complex and variable marine environment. In contrast, the controller introduced in this paper can interact iteratively with the ASV in real time in such environments, continuously learning the optimal control law from the generated state data, thus overcoming the limitations of relying on system model.

In recent years, inspired by biological systems, reinforcement learning (RL) has become an effective tool for addressing complex problems such as nonlinear dynamics, high-dimensional state spaces, and time-varying disturbances (Buşoniu et al., 2018; Cheng et al., 2023; Huang et al., 2025; Wang et al., 2019; Zhao et al., 2020). In the face of these problems, the RL as a direct adaptive control method, mainly focuses on decision-making in complex environments to minimize long-term costs, thereby achieving optimal control (Lewis and Vrabie, 2009; Modares and Lewis, 2014; Wang et al., 2024b; Yuan et al., 2022). As a result, the RL has shown significant potential for realizing optimal control of ASVs (Liu et al., 2024; Nguyen et al., 2023; Vu et al., 2022a; Wang et al., 2023). To address the optimal control problem of ASVs under external disturbance uncertainties, researchers have proposed an adaptive optimal control strategy to achieve optimal trajectory tracking for ASVs (Vu et al., 2022a). Considering the decoupling of the kinematic and dynamic subsystems in ASVs, an adaptive RL algorithm is combined with a kinematic controller to achieve precise trajectory tracking control (Vu et al., 2022b). In addition, some studies combine the kinematic and dynamic models of ASVs with formation control and RL to design control laws, thereby achieving optimal trajectory tracking for ASV formations (Nguyen et al., 2023). Although these studies provide robust control strategies, they often overlook the impact of system parameter variations on the optimal control strategy. The aforementioned methods have not sufficiently explored how to achieve optimal trajectory tracking control for ASVs when the system model is uncertain and environmental information is unknown, which is the core motivation behind this study. Meanwhile, unlike the aforementioned methods, this paper adopts a model-free data-driven RL method, which learns the control law directly from collected system data, thereby avoiding the influence of model parameters on control performance.

Inspired by the above discussion, we propose a model-free data-driven RL algorithm to address the optimal trajectory tracking control problem of ASVs under the influence of model uncertainties and external disturbances. The main contributions of this paper are as follows:

This paper introduces the H_∞ control framework enhances the robustness of the ASV system. Building on this, we propose an RL control algorithm based on the dynamic model of the ASV system. This algorithm effectively addresses the challenges of model uncertainties and disturbances in complex marine environments under specific operational conditions by incorporating dynamic model information of the ASV.

This paper overcomes the limitations of traditional model–dependent methods by proposing a model-free data-driven RL control algorithm. This algorithm eliminates reliance on prior information about the dynamic model of the ASV system and directly learns the optimal control law from online interactive data. Compared to existing control methods by Zhou et al. (2023a) and Hasanvand and Seif (2024), the proposed algorithm not only mitigates the impact of model uncertainties and disturbances on control performance but also enhances adaptability to complex environments, thereby improves feasibility for engineering deployment.

This paper is organized as follows. Section 2 provides the background and the problem description. Section 3 develops a model-based RL algorithm and a model-free data-driven RL algorithm for ASVs. Simulation and comparison results are presented in Section 4, and the conclusion is given in Section 5.

Preliminaries

The ASV model with 3 degrees of freedom can be represented as follows

{\begin{matrix} \overset{\cdot}{η} = R (ψ) ν \\ M \overset{\cdot}{ν} = - C (ν) ν - D (ν) ν + τ + d (t) \end{matrix}

(1)

where $η = {[x, y, ψ]}^{T}$ represents the position coordinates $(x, y)$ and the heading angle ψ, $ν = {[u, v, r]}^{T}$ represents the velocity vector, $τ = {[τ_{1}, τ_{2}, τ_{3}]}^{T}$ represents the designed control input and $d = {[d_{α}, d_{β}, d_{γ}]}^{T}$ represents the time-dependent environmental perturbations experienced at the sea surface. In equation (1), $R (ψ)$ represents the rotation matrix, M denotes the inertia matrix, $C (ν)$ is the Coriolis and centripetal matrix, and $D (ν)$ is the damping matrix. We define $M = Δ M + M_{0}$ , $C = Δ C + C_{0}$ , and $D = Δ D + D_{0}$ , where $M_{0}$ , $C_{0}$ , and $D_{0}$ are the nominal terms and $Δ M$ , $Δ C$ , and $Δ D$ denote the uncertainties. The specific parameters can be referred to in the study by Hou et al. (2024).

Letting $x_{1} = η$ and $x_{2} = Rν$ , the ASV system (1) can be expressed as follows

{\begin{matrix} \overset{\cdot}{x_{1}} = x_{2} \\ \overset{\cdot}{x_{2}} = f + gτ + h \end{matrix}

(2)

where $f = RSν - R M_{0}^{- 1} Cν - R M_{0}^{- 1} Dν$ , $g = R M_{0}^{- 1}$ and $h = R M_{0}^{- 1} Δ Cν + R M_{0}^{- 1} Δ Dν - R M_{0}^{- 1} d - R M_{0}^{- 1} Δ M R^{- 1} (\overset{\cdot}{x_{2}} - SRν)$ is the lumped uncertainties. Denoting $X = {[{x_{1}}^{T}, {x_{2}}^{T}]}^{T} \in ℝ^{6}$ , the ASV system (2) can be rewritten as

\begin{matrix} Ẋ = q_{1} G + q_{2} τ + q_{3} h \end{matrix}

(3)

where $q_{1} = [\begin{matrix} 1_{3 \times 3} & 0_{3 \times 3} \\ 0_{3 \times 3} & 1_{3 \times 3} \end{matrix}]$ , $G = [\begin{matrix} x_{2} \\ f \end{matrix}]$ , $q_{2} = [\begin{matrix} 0_{3 \times 3} \\ 1_{3 \times 3} \end{matrix}] g$ and $q_{3} = [\begin{matrix} 0_{3 \times 3} \\ 1_{3 \times 3} \end{matrix}]$ .

According to Modares et al. (2015) and from equation (3), we present the following Assumption 1 and Definition 1.

Assumption 1. There exists a Lipschitz continuous command generator function $k_{d} (\cdot) \in ℝ^{n}$ with $k_{d} (0) = 0$ such that

{\overset{\cdot}{η}}_{d} (t) = k_{d} (η_{d} (t))

(4)

where $η_{d} (t)$ is the bounded reference trajectory.

Definition 1 . The ASV system (3) is said to have $L_{2}$ -gain less than or equal to γ if the following lumped uncertainties attenuation condition is satisfied for all $h \in L_{2} [0, \infty)$

\int_{t}^{\infty} e^{- β (σ - t)} ∥ z ∥^{2} dϵ \leq γ^{2} \int_{t}^{\infty} e^{- β (σ - t)} ∥ h ∥^{2} dϵ

(5)

where $β > 0$ is the discount factor, $z = μ^{T} Qμ + τ^{T} Rτ$ with $μ = x_{1} - η_{d}$ and γ indicates the degree of attenuation from the lumped uncertainties input h to z .

This paper proposes an optimal trajectory tracking control law for ASVs in the presence of model uncertainties and external disturbances. Specifically, to address the challenges posed by uncertainties in the ASV model and disturbances, a model-free data-driven RL algorithm is proposed to learn the optimal control law, ensuring optimal trajectory tracking performance.

Main results

Model-based tracking control law

This subsection first addresses the lumped uncertainties using the H_∞ method and then designs a model-based RL algorithm to tackle the optimal trajectory tracking control problem for ASVs with model uncertainties and disturbances.

To mitigate the impact of the lumped uncertainties h on the ASV system (3), we employ the lumped uncertainties attenuation condition established in Definition 1. As given in equation (5), the impact of the lumped uncertainties h on the position tracking performance can be reduced by a certain degree at least equal to γ. The cost function is expressed as follows

V (X, t) = \int_{t}^{\infty} e^{- β (σ - t)} (μ^{T} Ψμ + τ^{T} Λτ - γ^{2} h^{T} h) dϵ

(6)

where Ψ and Λ are positive definite matrices. Then the value function is represented as

\begin{matrix} V^{*} (X, t) = & \min_{τ} \max_{h} \int_{t}^{\infty} e^{- β (σ - t)} (μ^{T} Ψμ + τ^{T} Λτ - γ^{2} h^{T} h) dϵ \end{matrix}

(7)

From the ASV system (3) and the cost function (6), the Hamilton–Jacobi–Isaacs (HJI) equation is derived as

\begin{matrix} H_{J} (V, τ, h) = & μ^{T} Ψμ + τ^{T} Λτ - γ^{2} h^{T} h - βV \\ + {\frac{∂V}{∂X}}^{T} (q_{1} G + q_{2} τ + q_{3} h) = 0 \end{matrix}

(8)

In fact, the control input τ and the lumped uncertainties h can be viewed as two players in a zero-sum differential game. According to the stationary conditions $\partial H_{J} ∕ ∂τ = 0$ and $\partial H_{J} ∕ ∂h = 0$ , the optimal control law $τ^{*}$ and lumped uncertainties $h^{*}$ are obtained as follows (Lewis et al., 2012)

\begin{matrix} τ^{*} & = - \frac{1}{2} Λ^{- 1} q_{2}^{T} \frac{\partial V^{*}}{∂X} \end{matrix}

(9)

\begin{matrix} h^{*} & = \frac{1}{2 γ^{2}} q_{3}^{T} \frac{\partial V^{*}}{∂X} \end{matrix}

(10)

Substituting equations (9) and (10) into equation (8), we obtain

\begin{matrix} H_{J} (V^{*}, τ^{*}, h^{*}) & = μ^{T} Ψμ - \frac{1}{4} {\frac{\partial V^{*}}{∂X}}^{T} q_{2} Λ^{- 1} q_{2}^{T} \frac{\partial V^{*}}{∂X} + \frac{1}{4 γ^{2}} \\ {\frac{\partial V^{*}}{∂X}}^{T} q_{3} q_{3}^{T} \frac{\partial V^{*}}{∂X} - β V^{*} + {\frac{\partial V^{*}}{∂X}}^{T} q_{1} G = 0 \end{matrix}

(11)

Based on the preceding discussion, we propose Theorem 1.

Theorem 1. Consider the ASV system (3). If the optimal control law $τ^{*}$ in equation (9) satisfies the lumped uncertainties attenuation condition (5), the system will be asymptotically stable when $h = 0$ and $β \leq 2 ∥ {(LΨ)}^{1 ∕ 2} ∥$ , where $L = q_{2} Λ^{- 1} q_{2}^{T} + q_{3} q_{3}^{T} ∕ (γ^{2})$ .

Proof. From equations (8)–(11), it can be shown that

\begin{matrix} H_{J} (V^{*}, τ, h) & = H_{J} (V^{*}, τ^{*}, h^{*}) + {(τ - τ^{*})}^{T} Λ (τ - τ^{*}) \\ - γ_{p}^{2} {(h - h^{*})}^{T} (h - h^{*}) . \end{matrix}

(12)

Substituting $τ = τ^{*}$ into equation (12) gives

\begin{matrix} H_{J} (V^{*}, τ^{*}, h) = - γ_{p}^{2} {(h - h^{*})}^{T} (h - h^{*}) \leq 0 \end{matrix}

(13)

Therefore, it can be concluded that

\begin{matrix} H_{J} (V^{*}, τ^{*}, h) = & μ^{T} Ψμ + {τ^{*}}^{T} Λ τ^{*} - γ^{2} h^{T} h - β V^{*} + {\overset{\cdot}{V}}^{*} \leq 0 \end{matrix}

(14)

where ${\overset{\cdot}{V}}^{*}$ represents the derivative of equation (7). From equation (14), it follows that

\begin{matrix} - β V^{*} + {\overset{\cdot}{V}}^{*} \leq - μ^{T} Ψμ - {τ^{*}}^{T} Λ τ^{*} + γ^{2} h^{T} h \end{matrix}

(15)

Multiplying both sides of equation (15) by $e^{- β (σ - t)}$ and integrating both sides, we get

\begin{matrix} e^{- βT} V^{*} (X (T)) - V^{*} (X (0)) \leq \int_{0}^{T} e^{- β (σ - t)} \\ (- μ^{T} Ψμ - {τ^{*}}^{T} Λ τ^{*}) dϵ + γ_{p}^{2} \int_{0}^{T} e^{- β (σ - t)} h^{T} hdϵ \end{matrix}

(16)

Since $V^{*} (\cdot) \geq 0$ , the inequality above implies

\begin{matrix} \int_{0}^{T} e^{- β (σ - t)} (μ^{T} Ψμ + {τ^{*}}^{T} Λ τ^{*}) dt \leq γ_{p}^{2} \int_{0}^{T} e^{- β (σ - t)} h^{T} hdϵ \end{matrix}

(17)

From equation (17), it is clear that the ASV system (3) meets the lumped uncertainties attenuation condition (5) when the optimal control law $τ^{*}$ from equation (9) is applied (Zhao et al., 2020). From Theorems 3 and 4 by Modares et al. (2015), the ASV system (3) is asymptotically stable as both β and h approach 0. For nonzero β, the ASV system (3) is locally asymptotically stable if $β \leq 2 ∥ {(LΨ)}^{1 ∕ 2} ∥$ , where $L = q_{2} Λ^{- 1} q_{2}^{T} + q_{3} q_{3}^{T} ∕ (γ^{2})$ . □

Remark 1. As demonstrated in the proof of Theorem 1, the control law we designed strictly satisfies the attenuation condition of H_∞ control defined in Definition 1. Specifically, this optimal control law effectively eliminates the effects caused by system uncertainties and external disturbances, ensuring that the ASV system maintains stability and robustness throughout its operation.

Based on equation (8), we propose the model-based RL algorithm for finding the solutions $τ^{*}$ and $h^{*}$ for equations (9) and (10).

Algorithm 1. Model-based RL algorithm
1: Initialize: $τ^{i} \leftarrow τ_{0}$ , $h^{i} \leftarrow h_{0}$ , $ϵ_{1} > 0$ and $ϵ_{2} > 0$
2: for $i = 1$ to n do
3: Substitute $τ^{i}$ and $h^{i}$ into $\begin{matrix} H_{J} (V^{i}, τ^{i}, h^{i}) = {(μ^{i})}^{T} Ψ μ^{i} + {(τ^{i})}^{T} Λ τ^{i} - γ^{2} {(h^{i})}^{T} h^{i} - β V^{i} \\ + {(\frac{\partial V^{i}}{\partial X})}^{T} (q_{1} G + q_{2} τ^{i} + q_{3} h_{i}) = 0 & (18) \end{matrix}$ to obtain $V^{i}$
4: Update control law and lumped uncertainties via $\begin{matrix} τ^{i + 1} & = - \frac{1}{2} Λ^{- 1} q_{2}^{T} \frac{\partial V^{i}}{\partial X} & (19) \end{matrix}$ $\begin{matrix} h^{i + 1} & = \frac{1}{2 γ^{2}} q_{3}^{T} \frac{\partial V^{i}}{\partial X} & (20) \end{matrix}$
5: if $∥ τ^{i + 1} - τ^{i} ∥ \leq ϵ_{1}$ and $∥ h^{i + 1} - h^{i} ∥ \leq ϵ_{2}$ then
6: Set $τ^{} \leftarrow τ^{i + 1}$ and $h^{} \leftarrow h^{i + 1}$
7: Break
8: else
9: Update parameters $τ \leftarrow τ^{i + 1}$ and $h \leftarrow h^{i + 1}$
10: end if
11: end for
12: Return the optimal control law $τ^{}$ and lumped uncertainties $h^{}$

Remark 2. The initial values $τ_{0}$ and $h_{0}$ in Algorithm 1 can be arbitrarily assigned. It should be emphasized that the successful implementation of Algorithm 1 depends on system model of the ASV. However, obtaining precise information about ASVs in complex disturbance environments remains challenging, which makes deploying ASVs in diverse and complex environments even more difficult.

Next, Lemma 1 is introduced to prove the convergence of Algorithm 1.

Lemma 1. Consider the ASV system (3) with the Assumption 1 being satisfied. The solution to equation (18) yields the cost function $V^{i}$ , where i represents the ith iteration. The control law and lumped uncertainties are derived by substituting the $V^{i}$ into equations (19) and (20). Then $V^{i}$ is refined by substituting the control law and lumped uncertainties into equation (18). Through this iterative optimization process, the optimal control law for the ASV system (3) is derived and converges to the control law specified by equations (9) and (10).

Proof. Lemma 1 has been rigorously proven by Wu and Luo (2012) and is thus omitted here. □

Data-driven tracking control law

This subsection adapts the model-based RL algorithm into a model-free data-driven RL algorithm, enabling optimal trajectory tracking control for the ASV system (3) without requiring a system model and external disturbance information.

Based on Algorithm 1, we propose Algorithm 2.

Algorithm 2. Model-free data-driven RL algorithm
1: Initialize: Collect historical data about the ASV system, $ϵ_{3} > 0$ and $ϵ_{4} > 0$
2: for $i = 1$ to n do
3: Using the data, solve for $V^{i }$ , $τ^{i }$ and $h^{i *}$ through $\begin{array}{r} e^{- β T} V^{i} (t + T) - V^{i} (t) = - \int_{t}^{t + T} e^{- β (σ - t)} (μ^{T} Ψ μ \\ + {(τ^{i})}^{T} Λ τ^{i} - γ^{2} {(h^{i})}^{T} h^{i}) d ϵ & (21) \\ - 2 \int_{t}^{t + T} e^{- β (σ - t)} {(τ^{i + 1})}^{T} Λ (τ - τ^{i}) d ϵ \\ + 2 γ^{2} \int_{t}^{t + T} e^{- β (σ - t)} {(h^{i + 1})}^{T} (h - h^{i}) d ϵ \end{array}$
4: if $∥ τ^{i + 1} - τ^{i} ∥ \leq ϵ_{3}$ and $∥ h^{i + 1} - h^{i} ∥ \leq ϵ_{4}$ then
5: Set $τ^{} \leftarrow τ^{i + 1}$ , $h^{} \leftarrow h^{i + 1}$
6: Break
7: else
8: Update parameters $τ^{i} \leftarrow τ^{i + 1}$ and $h^{i} \leftarrow h^{i + 1}$
9: end if
10: end for
11: Return the optimal control law $τ^{}$ and lumped uncertainties $h^{}$

Remark 3. The historical data mentioned in Algorithm 2 is obtained by applying a fixed control strategy and recording the ASV system data over a time interval of T. The control strategy only needs to be capable of stabilizing the ASV system. Then, without needing information about the ASV system and external information, the collected ASV system data are repeatedly applied to search for a series of updated control laws until the optimal control law is reached. In equation (21), τ and h represent historical data, while $τ^{i}$ , $τ^{i + 1}$ , $h^{i}$ , and $h^{i + 1}$ are approximated using NNs. Note that equation (21) can be solved simultaneously for both the value function $V^{i *}$ , the updated control law $τ^{i *}$ and lumped uncertainties $h^{i *}$ .

Remark 4. Given the universal approximation capability of NNs, we use NNs to implement Algorithm 2. During the implementation of Algorithm 2, the solutions $V^{i}$ , $τ^{i + 1}$ , and $h^{i + 1}$ of equation (21) are approximated by three radial basis function (RBF) NNs, which are given below

{\hat{V}}^{i} = W_{1}^{T} φ_{1} (Γ)

(22)

{\hat{τ}}^{i + 1} = W_{2}^{T} φ_{2} (Γ)

(23)

ĥ^{i + 1} = W_{3}^{T} φ_{3} (Γ)

(24)

where $Γ = {[x_{1}, x_{2}, μ, \overset{\cdot}{μ}]}^{T}$ , ${\hat{V}}^{i}$ , ${\hat{τ}}^{i + 1}$ , and $ĥ^{i + 1}$ are the approximated values, $W_{1} \in ℝ^{1 \times p_{1}}$ , $W_{2} \in ℝ^{3 \times p_{2}}$ , and $W_{3} \in ℝ^{3 \times p_{3}}$ are the weight matrices, where $p_{1}$ , $p_{2}$ and $p_{3}$ represent the number of neurons. $φ_{1} (Γ)$ , $φ_{2} (Γ)$ , and $φ_{3} (Γ)$ are the Gaussian basis functions. Substituting equations (22)–(24) into equation (21) results

\begin{matrix} e_{1} = & e^{- βσ} {\hat{V}}^{i + 1} - {\hat{V}}^{i} + \int_{t}^{t + T} e^{- β (σ - t)} (μ^{T} Ψμ \\ + τ^{T} Λτ - γ^{2} h^{T} h) dϵ \\ + 2 \int_{t}^{t + T} e^{- β (σ - t)} {({\hat{τ}}^{i + 1})}^{T} Λ (τ - {\hat{τ}}^{i}) dϵ \\ - 2 γ^{2} \int_{t}^{t + T} e^{- β (σ - t)} {(ĥ^{i + 1})}^{T} (h - ĥ^{i}) dϵ \end{matrix}

(25)

where $e_{1}$ is the Bellman approximation error, ${\hat{V}}^{i}$ , ${\hat{τ}}^{i}$ , and $ĥ^{i}$ are the output of the NNs. The weights of the NNs are iteratively updated using the least squares method until the stopping criteria of the algorithm are satisfied (Modares et al., 2015).

The model-free data-driven RL control algorithm is depicted in Figure 1. As seen in Figure 1, we use the collected data and apply the model-free data-driven RL control algorithm through iterative updates to ultimately acquire the optimal tracking control law, enabling optimal trajectory tracking control of the ASV system.

Figure 1.

The block diagram illustrating the proposed data-driven RL controller.

Next, Lemma 2 is introduced to demonstrate the convergence of Algorithm 2.

Lemma 2. Consider the ASV system (3) with the Assumption 1 being satisfied. If the optimal control law is obtained from equation (21), the solution will converge to the results given by equations (7), (9), and (10).

Proof. First, we prove the equivalence between Lemma 2 and Lemma 1. By differentiating equation (6), the following expression is obtained

{\overset{\cdot}{V}}^{i} = {\frac{\partial V^{i}}{∂X}}^{T} Ẋ

(26)

From equation (3), we obtain

Ẋ = q_{1} G + q_{2} τ^{i} + q_{3} h^{i} + q_{2} (τ - τ^{i}) + q_{3} (h - h^{i})

(27)

Substituting equations (18) and (27) into equation (26), we obtain

\begin{matrix} {\overset{\cdot}{V}}^{i} = & {\frac{\partial V^{i}}{∂X}}^{T} (q_{1} G + q_{2} τ^{i} + q_{3} h^{i} + q_{2} (τ - τ^{i}) + q_{3} (h - h^{i})) \\ = & β V^{i} - μ^{T} Ψμ - {(τ^{i})}^{T} Λ τ^{i} + γ^{2} {(h^{i})}^{T} h \\ + {\frac{\partial V^{i}}{∂X}}^{T} q_{2} (τ - τ^{i}) + {\frac{\partial V^{i}}{∂X}}^{T} q_{3} (h - h^{i}) \end{matrix}

(28)

From equations (19) and (20), it follows that

\begin{matrix} {\frac{\partial V^{i}}{∂X}}^{T} q_{2} & = - 2 {(τ^{i + 1})}^{T} Λ \end{matrix}

(29)

\begin{matrix} {\frac{\partial V^{i}}{∂X}}^{T} q_{3} & = 2 γ^{2} {(h^{i + 1})}^{T} \end{matrix}

(30)

Substituting equations (29) and (30) into equation (28) yields

\begin{matrix} {\overset{\cdot}{V}}^{i} = & β V^{i} - μ^{T} Ψμ - {(τ^{i})}^{T} Λ τ^{i} + γ^{2} {(h^{i})}^{T} h^{i} \\ - 2 {(τ^{i + 1})}^{T} Λ (τ - τ^{i}) + 2 γ^{2} {(h^{i + 1})}^{T} (h - h^{i}) \end{matrix}

(31)

Multiplying both sides of equation (31) by $e^{- β (σ - t)}$ and integrating over the interval $[t, t + T]$ , it follows that

\begin{matrix} \int_{t}^{t + T} e^{- β (σ - t)} {\overset{\cdot}{V}}^{i} dϵ = & \int_{t}^{t + T} e^{- β (σ - t)} (- μ^{T} Ψμ - {(τ^{i})}^{T} Λ τ^{i} \\ + γ^{2} {(h^{i})}^{T} h^{i} - 2 {(τ^{i + 1})}^{T} Λ (τ - τ^{i}) \\ + 2 γ^{2} {(h^{i + 1})}^{T} (h - h^{i})) dϵ \end{matrix}

(32)

Since $\int_{t}^{t + T} e^{- β (σ - t)} {\overset{\cdot}{V}}^{i} dϵ = e^{- βT} V^{i} (t + T) - V^{i} (t)$ , equation (21) follows directly from equation (32). It follows that the control law proposed in Lemma 2 converges to the optimal control law.

Next, a proof by contradiction is used to establish the uniqueness of the solution. Assume the existence of another solution $V_{a}^{i}$ that satisfies equation (31). Then, it follows that

\begin{matrix} {\overset{\cdot}{V}}_{a}^{i} = {\overset{\cdot}{V}}^{i} \end{matrix}

(33)

Since the derivatives of $V$ and $V_{a}^{i}$ are identical, it follows that the functions differ only by a constant. Integrating both sides gives

\begin{matrix} V_{a}^{i} - V^{i} = b \end{matrix}

(34)

Equation (6) shows that when the system is at the origin, its value is 0. Therefore, we have $V_{a}^{i} = V^{i} = 0$ , which implies that the assumed solution is the same as the known solution. This contradiction proves the uniqueness of the solution $V^{i}$ .

Lemma 2 is derived from Lemma 1 and has a unique solution, meaning that it is equivalent to Lemma 1. The proof is complete. □

Remark 5. It should be noted that the solution presented in Lemma 1 converges to $V^{*}$ , while the value $V^{i}$ obtained in Lemma 2 also converges to $V^{*}$ , as defined in equation (7). Since the optimal solution $V^{*}$ uniquely corresponds to the optimal $τ^{*}$ and $h^{*}$ , the solution $V^{i}$ also corresponds to the unique optimal $τ^{*}$ and $h^{*}$ .

Next, we propose Theorem 2 and prove that the stability of the ASV system is guaranteed by the control law derived from Algorithm 2.

Theorem 2. Consider the ASV system (3). If the optimal control law τ in equation (21) satisfies the lumped uncertainties attenuation condition (5), the system will be asymptotically stable when $h = 0$ and $β \leq 2 ∥ {(LΨ)}^{1 ∕ 2} ∥$ , where $L = q_{2} Λ^{- 1} q_{2}^{T} + q_{3} q_{3}^{T} ∕ (γ^{2})$ .

Proof. The control law in Theorem 2 is derived from equation (21) in Lemma 2. Since Lemma 2 is equivalent to Lemma 1, the control law in Lemma 2 is identical to the control law in Lemma 1. Subsequently, we can follow the proof method of Theorem 1 to establish Theorem 2, with the detailed derivation omitted here. □

Simulation results

In this section, we provide examples to illustrate the effectiveness of the designed model-free data-driven RL algorithm. We use the ASV called Cybership II, with its parameters detailed by Skjetne et al. (2005). The reference trajectory is defined as $η_{d} = {[5 \cos (0.25 t), 5 \sin (0.25 t), 0.25 t]}^{T}$ . The initial states are $η (0) = {[5.8, - 0.7, 0]}^{T}$ and $v (0) = {[0, 0, 0]}^{T}$ . Exogenous disturbances are $d = [\sin (0.1 t), \sin (0.1 t), and$ $\sin (0.1 t)]^{T}$ . The parameters of the HJI equation are selected as $Ψ = 98 I_{3}$ , $Λ = 3 I_{3}$ , and $γ^{2} = 5$ . The value of the time interval is selected to be $T = 0.05 s$ . The RBF NN uses the Gaussian function as the basis function. It is important to highlight that while the structure of the NN is not constrained, an adequate number of neurons is required to approximate the optimal control law. The convergence of the weights of the NNs is shown in Figure 2. One can observe from Figure 2 that the convergence is reached for all updated weights during the sixth iteration.

Figure 2.

Convergence of weights of NNs.

To emphasize the benefits of the RL method in Algorithm 2 introduced in this paper, we compare its performance with the RBF ASMC control method proposed by Jiang et al. (2022). In the comparison figure, the red line represents the reference trajectory, the blue line corresponds to the method proposed in this paper, and the green line represents the RBF ASMC method.

Figure 3 presents a comparison of the ASV’s desired trajectory tracking performance. The results demonstrate that the RL method can track the desired trajectory quickly and accurately, validating the effectiveness of Algorithm 2.

Figure 3.

Desired and actual trajectories compared with RBF ASMC in the $xy$ plane.

Figure 4 compares the tracking performance of the ASV for desired position trajectories. The x and y plots in Figure 4 show that the RL method achieves faster trajectory tracking under the same conditions, demonstrating superior response capability and control efficiency. The ψ plot in Figure 4 demonstrates that the RL method demonstrates tracking accuracy for the actual position trajectory. Figure 5 compares the position tracking errors between the RL method and the RBF ASMC method. Overall, the results in Figures 4 and 5 show that the RL method tracks the desired trajectory faster and demonstrates superior accuracy compared to the RBF ASMC method.

Figure 4.

Desired and actual state x, y, and ψ compared with RBF ASMC.

Figure 5.

Tracking errors $x_{e}$ , $y_{e}$ , and $ψ_{e}$ compared with RBF ASMC.

Figure 6 compares the tracking performance of the ASV desired velocity trajectories. The u and r plots in Figure 6 demonstrate that the RL method tracks the desired velocity more smoothly under identical conditions, significantly reducing fluctuations and errors in the tracking process, which in turn improves the system’s overall performance and reliability. The v plot in Figure 6 shows that the RL method achieves significantly faster tracking of the desired actual velocity than the RBF ASMC method. Figure 7 compares the RL and the RBF ASMC methods’ position tracking errors. In summary, the results in Figures 6 and 7 highlight the advantage of the RL method in achieving stable tracking of the desired velocity trajectories and demonstrate its faster response speed.

Figure 6.

Desired and actual state u, v, and r compared with RBF ASMC.

Figure 7.

Tracking errors $u_{e}$ , $v_{e}$ , and $r_{e}$ compared with RBF ASMC.

The comparisons outlined above emphasize the notable benefits of the RL method, especially with regard to important performance metrics such as tracking accuracy, stability, and response speed. These advantages not only highlight the potential of the RL method in practical applications but also emphasize its contribution to improving the system’s overall performance. Consequently, these results highlight the innovation and practical significance of our research, showing that the RL method effectively addresses the shortcomings of current methods.

Conclusion

This paper presents a model-free data-driven RL algorithm to address the optimal trajectory tracking control problem of ASVs in complex environments. The proposed algorithm eliminates the reliance on system models and environmental information. Instead, it learns the optimal control law directly from interaction data, effectively addressing the challenges posed by complex environments. Theoretical analysis shows that this method can yield the optimal control law. Future work will focus on the high-performance cooperative control of multiple ASVs in complex environments. In addition, we will explore control techniques for heterogeneous agents to overcome the reliance on system models and environmental information in traditional methods.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Natural Science Foundation of China (grant nos. 62173054 and 62073054).

ORCID iD

Yan Yan

Data availability statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

Buşoniu

De Schutter

Tolić

, et al. (2018) Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control 46: 8–28.

Cao

Peng

Liu

, et al. (2024) AUV collision avoidance strategy based on fuzzy reinforcement learning. IEEE Transactions on Intelligent Vehicles 99: 1–13.

Cao

Ren

Sun

(2022) Research on obstacle detection and avoidance of autonomous underwater vehicle based on forward-looking sonar. IEEE Transactions on Neural Networks and Learning Systems 34(11): 9198–9208.

Chen

(2013) Robust and H_∞ control. London: Springer.

Chen

Dai

Dong

(2024) Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems 35(6): 7520–7533.

Cheng

Liu

Wen

, et al. (2023) Data-driven time-varying formation-containment control for a heterogeneous air-ground vehicle team subject to active leaders and switching topologies. Automatica 153: 111029.

Wang

Peng

, et al. (2023) Safety-critical containment maneuvering of underactuated autonomous surface vehicles based on neurodynamic optimization with control barrier functions. IEEE Transactions on Neural Networks and Learning Systems 34(6): 2882–2895.

Hasanvand

Seif

(2024) Adaptive path-following control for displacement vessels at any loading conditions under ocean disturbances. Journal of Marine Science and Application 23: 900–913.

Hou

Yan

Jiang

, et al. (2024) Distributed finite-time sliding mode containment control for multi-unmanned surface vehicles with input and output quantization. Transactions of the Institute of Measurement and Control. Epub ahead of print October 22. DOI: 10.1177/01423312241273768.

10.

Huang

Peng

Zhang

, et al. (2024) Distributed optimal coordinated control for unmanned surface vehicles with interleaved periodic event-based mechanism. IEEE Transactions on Vehicular Technology 73(12): 18073–18086.

11.

Huang

Liu

Huang

, et al. (2020) Robust H_∞ control for nonlinear course system of unmanned surface vessel with polytopic uncertainty based on sum of squares. Transactions of the Institute of Measurement and Control 43(2): 390–399.

12.

Huang

Long

, et al. (2025) Observer-based human-in-the-loop optimal output cluster synchronization control for multiagent systems: A model-free reinforcement learning method. IEEE Transactions on Cybernetics 55(2): 649–660.

13.

Jiang

Yan

(2025) Adaptive predefined-time bounded consensus tracking control of multiagent systems under input/output quantization. IEEE Transactions on Systems, Man, and Cybernetics: Systems 99: 1–12.

14.

Jiang

Yan

, et al. (2022) Neural network based adaptive sliding mode tracking control of autonomous surface vehicles with input quantization and saturation. Ocean Engineering 265: 112505.

15.

Khalaji

Bahrami

(2022) Finite-time sliding mode control of underwater vehicles in 3D space. Transactions of the Institute of Measurement and Control 44(16): 3215–3228.

16.

Lewis

Vrabie

(2009) Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine 9(3): 32–50.

17.

Lewis

Vrabie

Syrmos

(2012) Optimal Control. Hoboken, NJ: Wiley.

18.

Chen

Che

(2024) Antiswing control of offshore cranes under ship rolling disturbances: An active disturbance rejection control based approach. Nonlinear Dynamics 112(23): 21097–21116.

19.

Liu

Yao

(2016) Robust Control: Theory and Applications. Hoboken, NJ: Wiley.

20.

Liu

Yan

Zhou

, et al. (2024) Event-triggered optimal tracking control for underactuated surface vessels via neural reinforcement learning. IEEE Transactions on Industrial Informatics 20(11): 12837–12847.

21.

Meng

Zhang

Han

, et al. (2024) Fault-tolerant control of underactuated unmanned surface vessels in presence of deception and injection attacks. Transactions of the Institute of Measurement and Control 47: 01423312241263386.

22.

Modares

Lewis

(2014) Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic Control 59(11): 3051–3056.

23.

Modares

Lewis

Jiang

(2015) H_∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems 26(10): 2550–2562.

24.

Lang

Fan

, et al. (2024) Trajectory tracking control of unmanned surface vehicle based on optimized barrier Lyapunov function under real ocean wave modeling. Journal of the Franklin Institute 361(16): 107182.

25.

Nguyen

Dang

Pham

, et al. (2023) Formation control scheme with reinforcement learning strategy for a group of multiple surface vehicles. International Journal of Robust and Nonlinear Control 34(3): 2252–2279.

26.

Qiao

Yin

Wang

, et al. (2023) Survey of deep learning for autonomous surface vehicles in marine environments. IEEE Transactions on Intelligent Transportation Systems 24(4): 3678–3701.

27.

Rigatos

(2023) A nonlinear optimal control approach for unmanned surface vessels. Marine Systems and Ocean Technology 18: 89–110.

28.

Skjetne

Fossen

Kokotović

(2005) Adaptive maneuvering, with experiments, for a model ship in a marine control laboratory. Automatica 41(2): 289–298.

29.

Pham

Dao

(2022a) Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels. ISA Transactions 130: 277–292.

30.

Tran

Pham

, et al. (2022b) Online actor-critic reinforcement learning control for uncertain surface vessel systems with external disturbances. International Journal of Control, Automation and Systems 20: 1029–1040.

31.

Wang

Dong

, et al. (2022a) Trajectory-tracking control of an underactuated unmanned surface vehicle based on quasi-infinite horizon model predictive control algorithm. Transactions of the Institute of Measurement and Control 44(14): 2709–2718.

32.

Wang

Yan

, et al. (2024a) Disturbance observer–based fixed-time sliding mode trajectory tracking control for marine surface vehicles with uncertain dynamics. Transactions of the Institute of Measurement and Control 46(8): 1452–1466.

33.

Wang

Xiao

Ren

, et al. (2024b) Formation cooperative trajectory tracking control for unmanned aerial vehicles via differential game and reinforcement learning. Transactions of the Institute of Measurement and Control 47: 1762–1770.

34.

Wang

Liu

, et al. (2023) Cooperative USV–UAV marine search and rescue with visual navigation and reinforcement learning-based control. ISA Transactions 137: 222–235.

35.

Wang

Sun

, et al. (2019) Deterministic policy gradient with integral compensator for robust quadrotor control. IEEE Transactions on Systems, Man, and Cybernetics: Systems 50(10): 3713–3725.

36.

Wang

(2022b) Prescribed performance trajectory tracking control of dynamic positioning ship under input saturation. Transactions of the Institute of Measurement and Control 44(1): 30–39.

37.

Wang

, et al. (2021) Sampled-data consensus of networked Euler-Lagrange systems: A discrete small-gain approach. IEEE Access 9: 156548–156555.

38.

Luo

(2012) Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H_∞ control. IEEE Transactions on Neural Networks and Learning Systems 23(12): 1884–1899.

39.

Yan

Yin

Hou

, et al. (2025) Adaptive fixed-time containment control of underactuated surface vessels with input quantization. Ocean Engineering 318: 120157.

40.

Yuan

Wang

Liu

, et al. (2022) Action mapping: A reinforcement learning method for constrained-input systems. IEEE Transactions on Neural Networks and Learning Systems 34(10): 7145–7157.

41.

Zhang

Chen

(2022) An adaptive tracking control method for offshore cranes with unknown gravity parameters. Ocean Engineering 260: 111809.

42.

Zhao

Liu

Lewis

(2020) Robust formation control for cooperative underactuated quadrotors via reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems 32(10): 4577–4587.

43.

Zhou

Shen

Nie

, et al. (2023a) Event-triggered adaptive dynamic programming for optimal tracking control of unmanned surface vessel with input constraints. Transactions of the Institute of Measurement and Control 45(15): 2835–2847.

44.

Zhou

Yan

, et al. (2023b) Event-triggered approximate optimal path-following control for unmanned surface vehicles with state constraints. IEEE Transactions on Neural Networks and Learning Systems 34(1): 104–118.