Automatic tuning of robust model predictive control in iterative tasks using efficient Bayesian optimization

Abstract

Robust model predictive control (RMPC) is an effective technology for controlling uncertain systems while robustly handling constraints, and its closed-loop performance heavily relies on the selection of objective functions. However, the objective functions are typically chosen to be close to the real control objectives, despite an objective function that leads to less conservative constraints often provides better closed-loop performance. In this paper, we propose an automatic tuning framework for RMPC in iterative tasks. In particular, we parameterize RMPC and develop a Bayesian optimization (BO) method to tune it by solving a black-box optimization problem. We then introduce an efficient transfer learning framework within BO, which speeds up the searching process and enhances the controller performance. The effectiveness of the proposed tuning framework is illustrated on numerical examples.

Keywords

Robust model predictive control Bayesian optimization iterative learning control

Introduction

Model predictive control (MPC) has attained remarkable success in recent decades, because of its disturbance rejection (Draeger et al., 1995) and constraint handling (Morari and Lee, 1999) capabilities. It has been widely applied in various fields (Darby and Nikolaou, 2012), such as robotics, process control and reinforcement learning (Chua et al., 2018; Kordabad et al., 2021; Pfrommer et al., 2022). However, the performance of the MPC controllers can be degraded by a series of factors, including an uncertain system model (Piga et al., 2019), a limited terminal set (Rosolia and Borrelli, 2017, 2018), or an inappropriate objective function (Marco et al., 2016). To resolve or compensate for these issues, most classic MPC controllers are designed manually and offline, which remains unchanged mainly during control. With the increasing computational power and the recent progress in machine learning, learning-based MPC methods, which exploit the data collected during operation to automatically design the MPC controllers online, have received more research interest.

Performing iterative tasks in control systems has been extensively studied in the literature (Rosolia et al., 2022; Wabersich and Zeilinger, 2022; Zhang et al., 2021), where one task execution is usually called an “iteration.” Iterative learning control (ILC) is an effective strategy for iterative tasks to achieve high closed-loop performance by learning from previous iterations. At each iteration, the system starts from the same initial condition and the goal is to track a given reference trajectory. Recently, a novel ILC controller based on MPC called iterative learning model predictive control (LMPC) (Rosolia and Borrelli, 2017, 2018) has been proposed without needing to know a given reference. The LMPC policy recursively constructs a terminal safe set from historical trajectories while updating a terminal cost function. Moreover, LMPC guarantees recursive feasibility and monotonic performance improvements at each iteration for a linear time-invariant (LTI) system (Rosolia and Borrelli, 2017). LMPC has been successfully applied in an autonomous racing example (Rosolia et al., 2017a). A detailed introduction to LMPC can be found in Rosolia and Borrelli (2018).

In the case of systems with unknown uncertainties, deterministic MPC schemes lose all the guarantees for constraint satisfaction. In contrast, robust model predictive control (RMPC) (Chisci et al., 2001; Mayne et al., 2005) is a powerful tool for handling uncertainties. A review of RMPC can be found in Saltik et al. (2018). Correspondingly, Ugo Rosolia proposed robust learning model predictive control (RLMPC) (Rosolia et al., 2017b, 2022) which robustly satisfies system constraints while iteratively improving the control performance. However, increasing the robustness of the LMPC controllers usually results in control performance degradation. Because the performance degradation in RLMPC is induced by the conservation of constraints, an efficient remedy is to revisit the hyperparameter tuning process (Garriga and Soroush, 2010), which has been proven effective in maximizing the closed-loop performance on a true system (Piga et al., 2019).

Bayesian optimization (BO) has recently been growing in popularity as a self-tuning strategy for controllers. BO is a powerful technique for optimizing complicated black-box functions (Brochu et al., 2010) and has been widely used for hyperparameter tuning in machine learning algorithms (Snoek et al., 2015). Compared with other derivative-free optimization algorithms, such as particle swarm optimization and genetic algorithms, BO is considerably faster and more data-efficient (Lu et al., 2020). BO has been adopted in a few MPC tuning problems (Guzman et al., 2022; Lu et al., 2020; Piga et al., 2019; Sorourifar et al., 2021; Stróżecki et al., 2021) to attain higher overall closed-loop performance. A review of BO can be found in Shahriari et al. (2016). However, to the best of our knowledge, there is still no research on using BO for tuning the parameters in LMPC while robustly satisfying constraints.

In this work, we propose a data-driven controller tuning framework that first considers robust constraint satisfaction when combining BO and MPC in iterative tasks. Given a LTI system with unknown uncertainties, our framework maximizes the integrated performance of RLMPC using BO while robustly satisfying all the constraints. Moreover, we propose an efficient scheme to better apply BO in iterative tasks. Considering the similarity of the datasets among the different iterations in LMPC, we introduce a transfer learning framework, which effectively extracts the information from previous iterations and speeds up the searching process.

The remainder of this paper is organized as follows. In the “Problem statement” section, we present the primary form of the robust optimal control problem and discuss the necessity of using BO. Next, we provide the fundamental background of RLMPC and BO in the “Technical background” section. Then, we introduce our method in the “RLMPC tuning using BO” section, including proposing a parameterized RLMPC problem, combining BO with RLMPC and introducing a transfer learning framework for RLMPC. An illustrative example and an ablation study are provided in the “Numerical example” section. Finally, the “Conclusion” section concludes this paper.

Problem statement

Given an initial state $x_{S}$ , we consider a system that follows an LTI model with unknown uncertainty

x_{t + 1} = A x_{t} + B u_{t} + w_{t}

(1)

where $x_{t} \in R^{n_{x}}$ and $u_{t} \in R^{n_{u}}$ are the state and input at time instant $t$ . $w_{t} \in W \subset R^{n_{x}}$ is the bounded uncertainty, where $W$ is assumed to be a compact convex set that contains the origin.

We consider that the state and input are subject to the constraints

x \in X, u \in U

(2)

where $X$ and $U$ are assumed to be compact polytopes.

We aim to optimize the performance of the following infinite horizon optimal control problem under unknown uncertainties

\begin{matrix} \min_{u_{0}, u_{1}, . . .} \sum_{t \geq 0} l (x_{t}, u_{t}) \\ s . t . x_{t + 1} = A x_{t} + B u_{t} + w_{t}, \\ x_{t} \in X, \\ u_{t} \in U, \\ x_{0} = x_{S}, \\ t = 0, 1, 2, . . ., \end{matrix}

(3)

where $l (x_{t}, u_{t}) = x_{t}^{T} Q x_{t} + u_{t}^{T} R u_{t}$ is a quadratic form of the stage cost. $Q$ and $R$ are positive definite diagonal weight matrices. Note that we select the linear quadratic function as our objective function due to its wide application in MPC and its effective representation for most control objectives.

Nonetheless, the optimal control problem (equation (3)) is usually hard to solve directly due to the infinite control horizon and the unknown uncertainty. In this paper, we use RLMPC to solve the problem in a finite-horizon fashion and improve its performance using BO.

Remark 1: In practice, each optimal control problem has a finite-time duration. It is common in the literature to adopt an infinite time formulation for simplicity. We follow such a form in this paper, and it does not affect the practicality of the proposed method.

Technical background

This section presents the technical background that provides the basis of RLMPC and BO.

RLMPC

Control policy approximation

A common approach to solve problems with bounded uncertainties is to decouple the dynamics (equation (1)) into a nominal state $s_{t}$ and an error state $e_{t} = x_{t} - s_{t}$ . An affine state-feedback control law for the actual control input $u_{t}$ can be given by Camacho and Bordons (2016)

u_{t} = K (x_{t} - s_{t}) + v_{t}

(4)

where $K \in R^{n_{u} \times n_{x}}$ is a feedback gain matrix, and $v_{t} \in R^{n_{u}}$ is an auxiliary control input. The nominal dynamics and the error system dynamics are given by

s_{t + 1} = A s_{t} + B v_{t}

(5)

e_{t + 1} = A_{k} e_{t} + w_{t}

(6)

where $A_{k} = A + BK$ . Therefore, the nominal system is disturbance-free, and the uncertainty $w_{t}$ only affects $e_{t}$ .

Since the real cost in equation (3) is difficult to optimize straightforwardly, we consider a nominal cost of the form $\sum_{t \geq 0} l (s_{t}, v_{t})$ .

As a result, given the feedback policy (equation (4)), the nominal dynamics (equation (5)) and the error dynamics (equation (6)), the robust approximation of equation (3) can be formulated as

\begin{matrix} \min_{v_{0}, v_{1}, . . .} \sum_{t \geq 0} l (s_{t}, v_{t}) \\ s . t . s_{t + 1} = A s_{t} + B v_{t}, \\ s_{t} \in X ⊖ S_{t} \\ v_{t} \in U ⊖ K S_{t} \\ s_{0} = x_{S} \\ t = 0, 1, 2, . . ., \end{matrix}

(7)

where $S_{t}$ is the error set, which is a compact polytope that contains all the possible values of $e_{t}$ at time instant $t$ . $S_{t}$ is defined as

S_{t} : = \sum_{i = 1}^{t} A_{K}^{i - 1} W

(8)

Subsequently, to turn problem (equation (7)) into a finite-horizon optimal problem, we introduce the notions of a convex sampled safe set and a terminal cost in RLMPC. We mainly borrow these following notions from Rosolia et al. (2017b). A more detailed introduction to LMPC theory can be referred to Rosolia and Borrelli (2017b).

Convex safe set

Consider an iterative task aiming to optimize the problem (equation (7)). At the $j^{th}$ iteration of the iterative task, let the input and state of the nominal system (equation (5)) be denoted as

v^{j} = [\begin{matrix} v_{0}^{j}, v_{1}^{j}, v_{2}^{j}, . . ., v_{t}^{j}, . . . \end{matrix}]

(9)

s^{j} = [\begin{matrix} s_{0}^{j}, s_{1}^{j}, s_{2}^{j}, . . ., s_{t}^{j}, . . . \end{matrix}]

(10)

Then, we can introduce the definition of the sampled safe set $S S^{j}$ as the terminal set

S S^{j} = {\cup_{i \in M^{j}} \overset{\infty}{\cup_{t = 0}} s_{t}^{i}}

(11)

where $M^{j} \subseteq {0, . . ., j}$ is the set of indices related to the successful iterations, that is, $S S^{j}$ collects all the nominal trajectories that converge to the origin up to the $j^{th}$ iteration.

Considering that a convex terminal set will significantly reduce the computational burden, we then try to construct a convex safe set, which is the convex hull of $S S^{j}$

\begin{matrix} C S^{j} = conv (S S^{j}) = \\ {\sum_{i = 1}^{| S S^{j} |} α_{i} z_{i} : α_{i} \geq 0, \sum_{i = 1}^{| S S^{j} |} α_{i} = 1, z_{i} \in S S^{j}} \end{matrix}

Terminal cost

The definition of the terminal cost is given based on the convex safe set $C S^{j}$ . At time $t$ of the $j^{th}$ iteration, the cost of the closed-loop trajectory (equation (10)) and input sequence (equation (9)) is defined as

V^{j} (s_{t}^{j}) = \sum_{k \geq 0} l (s_{t + k}^{j}, v_{t + k}^{j})

(12)

Then, we introduce the Barycentric function (Jones and Morari, 2010) as the terminal cost

P^{j} (s) : = {\begin{matrix} p^{j, *} (s) & if s \in C S^{j}, \\ + \infty & else \end{matrix}

(13)

where

\begin{matrix} p^{j, *} (s) = \min_{λ_{t} \geq 0, \forall t \in [0, \infty)} \sum_{k = 0}^{j} \sum_{t = 0}^{\infty} λ_{t}^{k} V^{k} (s_{t}^{k}) \\ s . t . \sum_{k = 0}^{j} \sum_{t = 0}^{\infty} λ_{t}^{k} = 1, \sum_{k = 0}^{j} \sum_{t = 0}^{\infty} λ_{t}^{k} s_{t}^{k} = s \end{matrix}

Accordingly, $P^{j} (s)$ assigns the minimum cost to each point within the terminal set. The cost is calculated based on the trajectories in $C S^{j}$ .

Finally, integrating the notions above, we present a RLMPC problem that solves the following finite-horizon optimal control problem

\begin{matrix} V_{N}^{j} (s_{t}^{j}) = \\ \min_{v_{t | t}^{j}, . . ., v_{t + N - 1 | t}^{j}} \sum_{k = t}^{t + N - 1} l (s_{k | t}^{j}, v_{k | t}^{j}) + P^{j - 1} (s_{t + N | t}^{j}) \\ s . t . s_{k + 1 | t}^{j} = {As}_{k | t}^{j} + {Bv}_{k | t}^{j}, \\ s_{k | t}^{j} \in X ⊖ S_{t}, \\ v_{k | t}^{j} \in U ⊖ K S_{t}, \\ s_{t | t}^{j} = s_{t}^{j}, s_{t + N | t}^{j} \in C S^{j}, \\ k = t, . . ., t + N - 1 \end{matrix}

(14)

While problem (equation (14)) is an alternative form for solving the problem (equation (3)), an inappropriate selection of the objective function in equation (7) might give rise to suboptimal performance. Therefore, we propose a BO-based tuning method for performance improvement in the “RLMPC tuning using BO” section.

BO

The principal idea of BO is to construct a surrogate model of the black-box objective function with a set of data. The fitted surrogate model provides a posterior distribution of the objective function, according to which an acquisition function is applied to determine where to sample next. Then, we add the newly sampled data to the dataset and update the surrogate model. This process is only terminated once the surrogate model converges to the optimum, or the maximum number of BO trade-off is reached.

Gaussian process

The Gaussian process (GP) (Williams and Rasmussen, 2006) model is the most commonly used surrogate model in BO. Given a mean function $m : R^{d} \to R$ and a covariance function $k : R^{d} \times R^{d} \to R$ , a $G P (m, k)$ is specified as a distribution over functions. Given an unknown objective $f (\cdot)$ and a set of initial $n$ noisy observations $y_{1 : n}, y_{i} = f (θ_{i}) + ϵ_{i}, ϵ_{i} ~ N (0, σ^{2}), i = 1, 2, . . ., n, σ \in R$ , the posterior distribution for any test point $θ$ is then presented as a Gaussian distribution $N (μ_{n} (θ), σ_{n}^{2} (θ))$ with mean and covariance

μ_{n} (θ) = k {(θ)}^{T} {(K + σ^{2} I_{n})}^{- 1} y_{1 : n}

(15)

σ_{n}^{2} (θ) = k (θ, θ) - k {(θ)}^{T} {(K + σ^{2} I_{n})}^{- 1} k (θ)

(16)

where $[K]_{i, j} = k (θ_{i}, θ_{j})$ , $[k (θ)]_{j} = k (θ, θ_{j})$ , and $I_{n}$ is the $n$ -dimensional identity matrix. In this paper, we select the covariance function $k$ to be the squared exponential function

k (θ, θ^{'}) = \exp (- \frac{1}{2 l^{2}} ∥ θ - θ^{'} ∥^{2})

(17)

where $l$ is the length-scale parameter.

Acquisition function

Based on the posterior functions (equations (15) and (16)), an acquisition function is then employed to search for the next sampled point. The acquisition function leads the sampled points to the optimum solution by exploration and exploitation. The acquisition functions can be classified as either using improvement-based criteria, such as probability of improvement (PI) (Kushner, 1964) and expected improvement (EI) (Mockus, 1998), or using confidence-based criteria, such as Gaussian process upper confidence bound (GP-UCB) (Srinivas et al., 2012). We choose GP-UCB as the acquisition function used in our work. At the $t^{th}$ trial, GP-UCB selects a point $θ_{t}$ while taking both the high predictive mean and high predictive variance into consideration, which is defined as

α_{t} (θ) = μ_{t - 1} (θ) + β_{t}^{\frac{1}{2}} σ_{t - 1} (θ)

(18)

θ_{t} = \underset{θ}{argmax} α_{t} (θ)

(19)

where $β_{t}$ are constants that determine the degree of exploration. The objective (equation (19)) prefers points $θ$ where $f$ is both uncertain (large $σ_{t - 1} (\cdot)$ ) and is expected to achieve high rewards (large $μ_{t - 1} (\cdot)$ ).

RLMPC tuning using BO

As we mentioned in the “Control policy approximation” section, the optimal control problem (equation (3)) is reformulated into a RLMPC form by decoupling the dynamics and introducing an approximated nominal cost. However, performance degradation still exists due to the inappropriate objective. In this section, we propose a parameter tuning framework that automatically adjusts the quadratic objective function and the conservatism of the constraints to achieve performance improvement.

Parameterized RLMPC

In regard to the inappropriate objective, we choose the weight matrices of the cost function $Q$ and $R$ to be parameterized with a vector $θ \in R^{n_{θ}}$ , that is

l (x_{t}, u_{t}, θ) = x_{t}^{T} Q (θ) x_{t} + u_{t}^{T} R (θ) u_{t}

(20)

Since $Q (θ)$ and $R (θ)$ are diagonal matrices, only the diagonal elements are tuned during BO.

Meanwhile, the terminal cost $P^{j - 1} (s)$ in RLMPC is also parameterized as $P^{j - 1} (s, θ)$ depending on $Q (θ)$ and $R (θ)$ .

The feedback gain $K$ generally has a major impact on the controller performance by determining the shape of tubes in RMPC, which reflects the extent of the constraint conservatism. Many previous works (Bujarbaruah et al., 2018; Kögel and Findeisen, 2020; Rosolia et al., 2017b) choose $K$ to be the optimal LQR gain for system (equation (5)) with the specified $Q$ and $R$ . We follow these settings and set $K (θ)$ with respect to $A$ , $B$ , $Q (θ)$ and $R (θ)$ , namely to be

K (θ) = R {(θ)}^{- 1} B^{T} P (θ)

(21)

where $P (θ)$ is the solution of algebraic Riccati equation

P (θ) A + A^{T} P (θ) - P (θ) BR {(θ)}^{- 1} B^{T} P (θ) + Q (θ) = 0

(22)

Correspondingly, the error set (equation (8)) is parameterized as

S_{t} = \sum_{i = 1}^{t} A_{K} {(θ)}^{i - 1} W

(23)

where $A_{K} (θ) = A + BK (θ)$ .

We reformulate the problem (equation (14)) after parameterization as

\begin{matrix} V_{N}^{j} (s_{t}^{j}, θ) = \\ \min_{v_{t | t}^{j}, . . ., v_{t + N - 1 | t}^{j}} \sum_{k = t}^{t + N - 1} l (s_{k | t}^{j}, v_{k | t}^{j}, θ) + P^{j - 1} (s_{t + N | t}^{j}, θ) \\ s . t . s_{k + 1 | t}^{j} = {As}_{k | t}^{j} + {Bv}_{k | t}^{j}, \\ s_{k | t}^{j} \in X ⊖ S_{t} (θ), \\ v_{k | t}^{j} \in U ⊖ K (θ) S_{t} (θ), \\ s_{t | t}^{j} = s_{t}^{j}, s_{t + N | t}^{j} \in C S^{j}, \\ k = t, . . ., t + N - 1 \end{matrix}

(24)

Thus, at the $j^{th}$ iteration, given a specified $θ$ and the initial state $x_{S}$ , we can recursively solve the problem (equation (24)) and obtain the nominal closed-loop trajectory $s^{j, θ} = [s_{0}^{j, θ}, s_{1}^{j, θ}, . . ., s_{t}^{j, θ}, . . .]$ as well as the control input sequence $v^{j, θ} = [v_{0}^{j, θ}, v_{1}^{j, θ}, . . ., v_{t}^{j, θ}, . . .]$ under $θ$ , where $s_{0}^{j, θ} = x_{S}$ . Substituting $s^{j, θ}$ and $v^{j, θ}$ into the real objective function $l (\cdot, \cdot)$ , we can obtain the real closed-loop performance $V^{j, θ} (x_{S}) = \sum_{k \geq 0} l (s_{k}^{j, θ}, v_{k}^{j, θ})$ as defined in equation (12). Thereby, we formulate a black-box function $f (\cdot)$ that builds the mapping from $θ$ to $V^{j, θ} (x_{S})$ , that is, $V^{j, θ} (x_{S}) = f (θ)$ . The optimum $θ$ that minimizes $V^{j, θ} (x_{S})$ will be determined by BO in our paper.

Remark 2: The real objective function has been defined in equation (3), where the weight matrices are set to be $Q$ and $R$ instead of $Q (θ)$ and $R (θ)$ .

Remark 3: Since the solutions of $s^{j, θ}$ and $v^{j, θ}$ are based on problem (equation (24)), the values of $s^{j, θ}$ , $v^{j, θ}$ and $V^{j, θ} (x_{S})$ all depend on the selection of $θ$ . However, given $s^{j, θ}$ and $v^{j, θ}$ , the calculation of $V^{j, θ} (x_{S})$ is completely based on the real objective function $l (\cdot, \cdot)$ , which is independent of $θ$ .

Generic BO for RLMPC

In this section, we introduce a fundamental framework that combines a generic BO with RLMPC for parameter tuning. The transfer learning method is introduced in the next section. A universal schematic that explains how BO works for RLMPC is shown in Figure 1.

Figure 1.

Illustration of how the BO method can be combined with RLMPC tuning. During each iteration, RLMPC receives the recommended optimal parameters from BO and computes the closed-loop performance repeatedly. At the end of the iteration, RLMPC chooses the optimal parameters from the current iteration and updates its sampled safe set.

Considering the $j^{th}$ iteration in RLMPC, the generic BO first randomly samples a set of $N$ points to construct an initial GP model. Then we, in addition, conduct $T$ trials to sample the points $θ_{1 : T}$ by our acquisition function (equation (18)). At each trial $i, 1 \leq i \leq T$ , we reformulate the parameterized RLMPC problem (equation (24)) with $θ_{i}$ and calculate the closed-loop performance $f (θ_{i})$ for $θ_{i}$ as $f (θ_{i}) = V^{j, θ_{i}} (x_{S})$ .

Finally, we choose the point with the minimum response value as the optimal point $θ^{*}$ for iteration $j$ and construct the convex sampled safe set by the nominal trajectories under $θ^{*}$ . This process is outlined in Algorithm 1.

Algorithm 1: Generic BO for RLMPC at iteration $j$
Input: sampled safe set $C S^{j - 1}$ , number of trials $T$ , coefficient $β_{t}$ for GP-UCB, initial state $x_{S}$
Output: updated sampled safe set $C S^{j}$
1 Randomly sample $N$ data points to construct the GP surrogate model $S_{0}$ .
2 for $t = 1$ to $T$ do
3 Choose the next optimal data point $θ_{t}$ based on equations (18) and (19).
4 Reformulate the parameterized RLMPC problem (equation (24)).
5 Perform the closed-loop experiment and compute the overall cost $V^{j, θ_{t}} (x_{S})$ .
6 $f (θ_{t}) \leftarrow V^{j, θ_{t}} (x_{S})$ .
7 Augment the training set with $(θ_{t}, f (θ_{t}))$ .
8 Update the surrogate model.
9 end
10 Select a point $θ$ with the minimum response value $f (θ)$ .
11 Construct the next convex sampled safe set $C S^{j}$ with the nominal trajectories under the optimal point $θ$ .

Efficient BO

A major drawback when using generic BO is that the sampled data from previous iterations are discarded, which results in a loss of data efficiency. In this section, we introduce efficient BO, a transfer learning method for BO that further improves the data efficiency and controller performance.

In the RLMPC problem (equation (14)), the only difference among different iterations is the scale of convex sampled safe sets. Intuitively, the datasets across the various iterations may have certain similar features, so that transfer learning can be utilized to obtain more information from the former iterations for the generic BO.

A simple and direct way to implement transfer learning is to reserve the historical data from previous iterations and use them to construct the surrogate model for current iteration. However, due to the variation of terminal sets, the same data point over different iterations may experience a massive discrepancy in its responsive value, which brings about adverse effects in transfer learning.

Nevertheless, different iterations may produce similar-looking functions of response values. We display a toy example in Figure 2 to verify and show that the overall shape of the response surface varies minimally between different iterations as the response surface obtains a similar level of improvement while updating the sampled safe set. Accordingly, it is suitable to introduce an efficient transfer learning framework proposed by Yogatama and Mann (2014), which fits a surrogate GP model using the normalized response values instead of the original ones. Specifically, at the $j^{th}$ iteration, the normalized response value for a trial $t$ is calculated as

y_{t}^{j} = \frac{f (θ_{t}) - μ^{j}}{σ^{j}}

(25)

where $μ^{j}$ and $σ^{j}$ are the mean and standard deviation of dataset, respectively, at the $j^{th}$ iteration. Since $μ^{j}$ and $σ^{j}$ are unknown, we estimate them by

{\hat{μ}}^{j} = \frac{1}{t} \sum_{t} f (θ_{t})

(26)

{\hat{σ}}^{j} = \sqrt{\frac{1}{t} \sum_{t} {(f (θ_{t}) - {\hat{μ}}^{j})}^{2}}

(27)

Therefore, efficient BO constructs the initial surrogate model using the data points with normalized response values from previous iterations instead of randomly sampling points like generic BO. Compared with generic BO, the initial surrogate model in efficient BO contains more useful data and is closer to the actual black-box function. As a result, efficient BO is more conducive to finding the optimal $θ$ faster and more accurately. We will validate its performance in our experiments.

Figure 2.

A toy example of response surfaces and the variation in the overall response value versus different iterations. (a) and (b) The response surfaces of iterations 1 and 2. (c) The variation of response value between these two iterations, which is the response value of iteration 2 minus the response value of iteration 1. It shows that the change of response value in most areas is close, demonstrating that the response surface is approximately in the translation and thus keeps a similar looking between two iterations.

As to specific settings of efficient BO, we set an upper bound on the size of the training set in case too much data slows down the GP fitting process and the earliest data points degrade the overall performance. When choosing an optimal point, we only select from the data points sampled during the current iteration.

Numerical example

In this section, we apply the proposed efficient BO for RLMPC to solve the following optimal control problem

\begin{matrix} V^{j} (x_{S}) = \min_{u_{0}^{j}, u_{1}^{j}, . . .} \sum_{t \geq 0} {x_{t}^{j}}^{T} {Qx}_{t}^{j} + {u_{t}^{j}}^{T} {Ru}_{t}^{j} \\ s . t . x_{t + 1}^{j} = {Ax}_{t}^{j} + {Bu}_{t}^{j} (x_{t}^{j}) + w_{t}^{j}, \\ x_{t}^{j} \in X, u_{t}^{j} \in U, \forall w_{t}^{j} \in W, \\ x_{0}^{j} = x_{S}, t = 0, 1, 2, . . . . \end{matrix}

(28)

We introduce two examples to illustrate the effectiveness of efficient BO. All of the experiments were carried out using Python 3.9.13. The optimization problems in the RLMPC are solved with the Ipopt solver, which is wrapped in package CasADi 3.5.5. Considering the randomness in BO, we run each algorithm six times independently with different random seeds.

L2 regulator

First, we consider an example of L2 regulator, where the linear system is given as

A = [\begin{matrix} 1.2 & 1.5 \\ 0 & 1.3 \end{matrix}], B = [\begin{matrix} 0 \\ 1 \end{matrix}]

(29)

The constraints of state and control input are given by $X = [- 10, 10] \times [- 10, 10]$ and $U = [- 1, 1]$ , respectively. The uncertainty is bounded within the set $W = [- 0.03, 0.03]$ .

We implement RLMPC with a control horizon $N = 3$ . The real weight matrices $Q$ and $R$ are $2 I_{2}$ and $I_{1}$ , respectively. We steer the system from $x_{S} = [4.0; 1.0]$ to the origin. At the $j^{th}$ iteration, RLMPC terminates at step $t$ when $V_{N}^{j} (s_{t}^{j}) \leq 10^{- 6}$ , where $s_{t}^{j}$ is the nominal state corresponding to $x_{t}^{j}$ . We run RLMPC for 30 iterations.

For efficient BO, we set the number of trials $T = 10$ for each iteration except for the first iteration, where we randomly sample 10 points to construct an initial surrogate GP model. The number of parameters that need to be tuned is 3, including the diagonal elements in $Q$ and $R$ . Let the tuned parameters be $θ_{L} = [θ_{L}^{1}, θ_{L}^{2}, θ_{L}^{3}]$ , then, $Q (θ)$ and $R (θ)$ are defined as

Q (θ_{L}) = [\begin{matrix} 2 θ_{L}^{1} & 0 \\ 0 & 2 θ_{L}^{2} \end{matrix}], R (θ_{L}) = [\begin{matrix} θ_{L}^{3} \end{matrix}]

(30)

Each tuned parameter $θ_{L}^{i}, i = 1, 2, 3$ can vary in $[1.0, 100.0]$ . Once the training set is more than 50, we reserve the most recently added 50 elements and discard the others.

The average of the current best performance for RLMPC and the efficient BO is shown in Table 1. The bold number in their respective columns denotes the iteration at which they reach the minimum. It can be seen that joined with the proposed efficient BO method, the performance of RLMPC receives a significant improvement from $837.03$ to $781.50$ .

Table 1.

Comparison on the average of the current best performance between RLMPC and efficient BO.

Iterations	RLMPC	Efficient BO	Iterations	RLMPC	Efficient BO
1	839.71	839.71	16	837.03	784.35
2	839.36	798.65	17	837.03	783.12
3	838.44	791.30	18	837.03	782.56
4	837.11	789.26	19	837.03	782.56
5	837.05	789.19	20	837.03	782.43
6	837.04	789.19	21	837.03	782.43
7	837.04	789.09	22	837.03	782.43
8	837.04	787.89	23	837.03	782.43
9	837.04	787.89	24	837.03	781.68
10	837.04	787.78	25	837.03	781.68
11	837.04	787.78	26	837.03	781.64
12	837.04	786.79	27	837.03	$781.50$
13	837.04	785.85	28	837.03	781.50
14	837.04	785.57	29	837.03	781.50
15	$837.03$	785.15	30	837.03	781.50

The bold number in their respective columns denotes the iteration at which they reach the minimum.

Continuous stirred tank reactor

The other example we consider is the linearized continuous stirred tank reactor (CSTR) with an exothermic irreversible first-order reaction $S \to P$ (Henson and Seborg, 1997). The model consists of two states, the concentration of reactant $C_{S}$ and the temperature of the reactor $T$ . The control input is the coolant stream $T_{C}$ .

The linearized CSTR system is given based on dos Reis de Souza et al. (2022), where

A = [\begin{matrix} 0.745 & - 0.002 \\ 5.610 & 0.780 \end{matrix}], B = [\begin{matrix} 5.6 \times 10^{- 6} \\ 0.464 \end{matrix}]

(31)

The state and control input are bound as $X = [- 2, 2] \times [- 10, 5]$ and $U = [- 4.5, 4.5]$ . The uncertainty set is given by $W = [- 0.1, 0.1] \times [- 0.2, 0.2]$ .

In this example, the control horizon $N$ is 3. The weight matrices $Q$ and $R$ are set to be $2 I_{2}$ and $I_{1}$ , respectively. The system is initialized with $x_{S} = [- 1.0; - 5.0]$ . RLMPC is terminated at the $j^{th}$ iteration whenever $V_{N}^{j} (s_{t}^{j}) \leq 10^{- 6}$ , where $t$ is the time step. The total number of RLMPC iterations is 20.

The number of trials $T$ for efficient BO is 2 at each iteration. Similarly, we only randomly sample 2 points to construct an initial surrogate GP model at the first iteration. Like in the L2 regulator example, we select the diagonal elements in $Q$ and $R$ as the tuned parameters, which are defined as $θ_{C} = [θ_{C}^{1}, θ_{C}^{2}, θ_{C}^{3}]$ . The parameter space is ${[1.0, 100.0]}^{3}$ . The maximum size of the training set is 10.

We list the current best performance of RLMPC and the efficient BO in Table 2, and we have bolded the minimum values for the respective columns. It shows that efficient BO improves the performance of RLMPC from 955.62 to 759.06.

Table 2.

Comparison on the average of the current best performance between RLMPC and efficient BO.

Iterations	RLMPC	Efficient BO	Iterations	RLMPC	Efficient BO
1	955.92	955.92	11	955.62	772.27
2	$955.62$	911.46	12	955.62	772.27
3	955.62	840.46	13	955.62	772.27
4	955.62	824.95	14	955.62	760.26
5	955.62	811.13	15	955.62	759.73
6	955.62	809.27	16	955.62	759.73
7	955.62	800.36	17	955.62	759.73
8	955.62	800.22	18	955.62	759.41
9	955.62	789.66	19	955.62	759.17
10	955.62	777.41	20	955.62	759.06

The bold number in their respective columns denotes the iteration at which they reach the minimum.

As we demonstrate in the “Parameterized RLMPC” section, under different selections of $θ$ , the degree of conservatism of the constraints also varies along with the objective function. Efficient BO makes certain constraints less conservative by adjusting $θ$ , so that state and control input trajectories with smaller real cost can be obtained, explaining the results of the above two examples. We further elaborate on it in the “Effectiveness analysis of efficient BO” section.

Effectiveness analysis of efficient BO

Figure 3 compares the state trajectory of RLMPC with that of efficient BO. Figure 3(a) and (b) shows the trajectories of L2 regulator and CSTR, respectively. The controller adjusted by efficient BO clearly deviates and finds a shorter path leading to the origin, resulting in the decrease in cost.

Figure 3.

Comparison of the state trajectory between RLMPC (yellow lines) and efficient BO (purple dashed lines). The yellow and purple solid points indicate the nominal states of RLMPC and the efficient BO, respectively: (a) L2 regulator and (b) CSTR.

Furthermore, we plot the state and control input tubes to illustrate how efficient BO can outperform RLMPC, as tubes intuitively reflect the conservatism of the constraints. The shapes of the state tubes are shown in Figure 4, where Figure 4(a) denotes the state tubes of L2 regulator and Figure 4(b) denotes the state tubes of CSTR. The input tubes of L2 regulator and CSTR are shown separately in Figures 5 and 6. Figure 4 shows that efficient BO enlarges the state tubes, which results in more conservative state constraints. In contrast, Figures 5 and 6 indicate that the size of the input tubes of efficient BO is clearly smaller than those of RLMPC (we only compare the first five input tubes), providing the controller with more input selections. The tubes as well as the nominal state and control input trajectories in Figures 4 –6 plot the range of possible real state and control input under disturbances. Therefore, to make the real state and control input satisfy the constraints, the larger the tubes, the smaller the selectable range of the nominal state and control input. For example, in Figure 6, the tube of the first control input of efficient BO is obviously smaller than that of RLMPC so that efficient BO can choose a larger nominal control input, and the same is true for the second to fifth nominal control inputs. Because of the ability to choose larger actions, efficient BO can obtain a shorter nominal state trajectory as shown in Figure 3. Meanwhile, although the state tubes of efficient BO are enlarged and efficient BO can only select states closer to the origin, the shortened nominal state trajectories can still robustly guarantee the satisfaction of all constraints. In these two examples, efficient BO tightens the state constraints while loosening the input constraints. Although the larger input partially increases the cost, the corresponding shorter state trajectory reduces the cost by far more than the increased part. As a result, the overall cost is significantly reduced, which explains the results in Tables 1 and 2.

Figure 4.

Comparison of the state tubes between RLMPC (yellow polytopes) and the efficient BO (purple polytopes). We only plot the first three state tubes of CSTR for clear.

Figure 5.

Comparison of the control input tubes between RLMPC (yellow error bar) and efficient BO (purple error bar) in L2 regulator. The length of the error bars indicates the size of the input tubes. The yellow and purple solid points indicate the nominal control inputs of RLMPC and efficient BO, respectively. The gray lines connecting these two subgraphs are used to clearly compare the length of the tubes.

Figure 6.

Comparison of the control input tubes between RLMPC (yellow error bar) and efficient BO (purple error bar) in CSTR. The meaning of each line is the same as that in Figure 5.

Comparison between the efficient BO and the other baselines

We compare the proposed efficient BO with several baselines, which can be regarded as an ablation study. The baselines are as follows:

Generic BO: A combination of RLMPC and BO without transfer learning, proposed in the “Generic BO for RLMPC” section.

Unnormalized efficient BO: A transfer learning method that constructs the GP surrogate model without using normalized response values as the efficient BO.

Unlimited efficient BO: A transfer learning method that constructs the GP surrogate model without an upper bound on the size of the training set like in the efficient BO.

Figure 7 plots the average of the current best performance with the standard error of the efficient BO and the three baselines. The results of L2 regulator are shown in Figure 7(a). The efficient BO searches faster after the fifteenth iteration with less variance. The efficient BO provides an average performance improvement of 7.26% from 785.44 to 781.50 compared with the generic BO. Meanwhile, the efficient BO performs 4.49% and 5.49% better than the unnormalized efficient BO from 784.00 to 781.50, and the unlimited efficient BO from 784.53 to 781.50, on average.

Figure 7.

Average of the current lowest cost of the efficient BO, generic BO, unnormalized efficient BO and unlimited efficient BO. The shaped region represents ±1 standard deviation: (a) L2 regulator and (b) CSTR.

Similarly, the results of CSTR in Figure 7(b) show that efficient BO averagely outperforms 6.63% than generic BO from 771.29 to 759.06. In addition, efficient BO achieves an improvement of 3.65% compared with the performance of unnormalized efficient BO 765.99. The improvement of efficient BO with respect to unlimited efficient BO is 10.54%, from 777.82 to 759.06.

As we have discussed in the “Effectiveness analysis of efficient BO” section, the lower the cost, the better $θ$ can be found so that the length of the state trajectory can be shortened as much as possible while limiting the control input size. Thus, the performance improvement of efficient BO shows that efficient BO is more capable of exploring to find a better $θ$ . Likewise, it indicates that efficient BO constructs a more accurate surrogate model than the other three baselines.

Conclusion

In this paper, we propose a framework that automatically tunes the parameters in RMPC to attain performance improvement for iterative tasks. In particular, we formulate the parameter tuning as a black-box optimization problem, where we use BO to find the optimal objective function and the optimal shape of RLMPC tubes. Moreover, regarding the connections among different iterations, we introduce a transfer learning framework for BO, the efficient BO, which has a higher searching speed and enhanced performance. We demonstrate the effectiveness of the proposed framework using two examples. Meanwhile, we compare the shape of tubes to more intuitively demonstrate how BO works on parameter tuning. Finally, we conduct an ablation study to validate the advantages of efficient BO.

The application of existing methods to high-dimensional systems is still limited since the computational burden of GP increases rapidly with the dimension. Moreover, high-dimensional GP needs far more data to construct an accurate surrogate model, which is usually challenging to achieve in practical settings. Future research will consider making use of improved GP techniques, for example, dynamic sparse GPs (Hewing et al., 2020), which relieve the above issues and better apply efficient BO to various domains.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Key Research & Development Program of China (No. 2021YFC1809003).

ORCID iD

Junbo Tong

References

Brochu

Cora

de Freitas

(2010) A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR. Available at: http://arxiv.org/abs/1012.2599

Bujarbaruah

Zhang

Tseng

, et al. (2018) Adaptive MPC for autonomous lane keeping. Available at: https://arxiv.org/abs/1806.04335

Camacho

Bordons

(2016) Model Predictive Control: Classical, Robust and Stochastic. Cham, Switzerland: Springer.

Chisci

Rossiter

Zappa

(2001) Systems with persistent disturbances: Predictive control with restricted constraints. Automatica 37(7): 1019–1028.

Chua

Calandra

McAllister

, et al. (2018) Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Bengio

Wallach

Larochelle

, et al. (eds) Advances in Neural Information Processing Systems (Vol. 31). New York: Curran Associates. Available at: https://proceedings.neurips.cc/paper_files/paper/2018/file/3de568f8597b94bda53149c7d7f5958c-Paper.pdf (accessed 4 August 2023).

Darby

Nikolaou

(2012) MPC: Current practice and challenges. Control Engineering Practice 20(4): 328–342.

dos Reis de Souza

Efimov

Raïssi

, et al. (2022) Robust output feedback model predictive control for constrained linear systems via interval observers. Automatica 135: 109951.

Draeger

Engell

Ranke

(1995) Model predictive control using neural networks. IEEE Control Systems Magazine 15(5): 61–66.

Garriga

Soroush

(2010) Model predictive control tuning methods: A review. Industrial & Engineering Chemistry Research 49(8): 3505–3515.

10.

Guzman

Oliveira

Ramos

(2022) Bayesian optimisation for robust model predictive control under model parameter uncertainty. Available at: https://doi.org/10.48550/arXiv.2203.00551

11.

Henson

Seborg

(1997) Nonlinear Process Control. Upper Saddle River, NJ: Prentice Hall.

12.

Hewing

Kabzan

Zeilinger

(2020) Cautious model predictive control using Gaussian process regression. IEEE Transactions on Control Systems Technology 28(6): 2736–2743.

13.

Jones

Morari

(2010) Polytopic approximation of explicit model predictive controllers. IEEE Transactions on Automatic Control 55(11): 2542–2553.

14.

Kögel

Findeisen

(2020) Fusing multiple time varying tubes for robust MPC. IFAC-PapersOnLine 53(2): 7055–7062.

15.

Kordabad

Esfahani

Lekkas

, et al. (2021) Reinforcement learning based on scenario-tree MPC for ASVs. In: 2021 American Control Conference (ACC), 25–28 May 2021. New Orleans, LA: IEEE, pp. 1985–1990.

16.

Kushner

(1964) A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering 86(1): 97–106.

17.

Kumar

Zavala

(2020) MPC controller tuning using Bayesian optimization techniques. Available at: https://doi.org/10.48550/arXiv.2009.14175

18.

Marco

Hennig

Bohg

, et al. (2016) Automatic LQR tuning based on Gaussian process global optimization. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), 16-21 May 2016. Stockholm, Sweden: IEEE, pp. 270–277.

19.

Mayne

Seron

Raković

(2005) Robust model predictive control of constrained linear systems with bounded disturbances. Automatica 41(2): 219–224.

20.

Mockus

(1998) The application of Bayesian methods for seeking the extremum. In: Mockus

Tiesis

Zilinskas

(eds) Towards Global Optimization 2. Amsterdam: North-Holland, pp. 117–129.

21.

Morari

Lee

(1999) Model predictive control: Past, present and future. Computers & Chemical Engineering 23(4): 667–682.

22.

Pfrommer

Gautam

Zhou

, et al. (2022) Safe reinforcement learning with chance-constrained model predictive control. In: Firoozi

Mehr

Yel

, et al. (eds) Proceedings of the 4th Annual Learning for Dynamics and Control Conference Proceedings of Machine Learning Research (Vol. 168). PMLR, pp. 291–303. Available at: https://proceedings.mlr.press/v168/pfrommer22a.html

23.

Piga

Forgione

Formentin

, et al. (2019) Performance-oriented model learning for data-driven MPC design. IEEE Control Systems Letters 3(3): 577–582.

24.

Rosolia

Borrelli

(2017) Learning model predictive control for iterative tasks: A computationally efficient approach for linear system. IFAC-PapersOnLine 50(1): 3142–3147.

25.

Rosolia

Borrelli

(2018) Learning model predictive control for iterative tasks: A data-driven control framework. IEEE Transactions on Automatic Control 63(7): 1883–1896.

26.

Rosolia

Carvalho

Borrelli

(2017a) Autonomous racing using learning model predictive control. In: 2017 American Control Conference (ACC), 24–26 May 2017. Seattle, WA: IEEE, pp. 5115–5120.

27.

Rosolia

Zhang

Borrelli

(2017b) Robust learning model predictive control for iterative tasks: Learning from experience. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC), 12–15 December 2017. Melbourne, VIC, Australia: IEEE, pp. 1157–1162.

28.

Rosolia

Zhang

Borrelli

(2022) Robust learning model-predictive control for linear systems performing iterative tasks. IEEE Transactions on Automatic Control 67(2): 856–869.

29.

Saltik

Özkan

Ludlage

, et al. (2018) An outlook on robust model predictive control algorithms: Reflections on performance and computational aspects. Journal of Process Control 61: 77–102.

30.

Shahriari

Swersky

Wang

, et al. (2016) Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104(1): 148–175.

31.

Snoek

Rippel

Swersky

, et al. (2015) Scalable Bayesian optimization using deep neural networks In: Bach

Blei

(eds) Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research (Vol. 37). Lille, France: PMLR, pp. 2171–2180. Available at: https://proceedings.mlr.press/v37/snoek15.html

32.

Sorourifar

Makrygirgos

Mesbah

, et al. (2021) A data-driven automatic tuning method for MPC under uncertainty using constrained Bayesian optimization. IFAC-PapersOnLine 54(3): 243–250.

33.

Srinivas

Krause

Kakade

, et al. (2012) Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory 58(5): 3250–3265.

34.

Stróżecki

Oufroukh

Kebbati

, et al. (2021) Automatic tuning of MPC for autonomous vehicle using Bayesian optimization. In: 2021 IEEE International Conference on Networking, Sensing and Control (ICNSC) (Vol. 1), 3–5 December 2021. Xiamen, China: IEEE, pp. 1–6.

35.

Wabersich

Zeilinger

(2022) Cautious Bayesian MPC: Regret analysis and bounds on the number of unsafe learning episodes. IEEE Transactions on Automatic Control 1: 8.

36.

Williams

Rasmussen

(2006) Gaussian Processes for Machine Learning (Vol. 2). Cambridge, MA: MIT Press.

37.

Yogatama

Mann

(2014) Efficient transfer learning method for automatic hyperparameter tuning. In: Kaski

Corander

(eds) Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics proceedings of Machine Learning Research (Vol. 33), 22–25 April 2014. Reykjavik, Iceland: PMLR, pp. 1077–1085. Available at: https://proceedings.mlr.press/v33/yogatama14.html

38.

Zhang

Wang

Masayoshi

(2021) Neural-network-based iterative learning control for multiple tasks. IEEE Transactions on Neural Networks and Learning Systems 32(9): 4178–4190.