Active nonlinear vibration control of a buckled beam based on deep reinforcement learning

Abstract

Vibration control in civil engineering is often challenging due to the nonlinear nature of structures. Traditional control strategies have limitations in terms of modeling accuracy and scalability, especially when analyzing complex nonlinear systems. To solve this problem, this study proposes a model-free active vibration control technique specifically for nonlinear systems, which employs deep reinforcement learning (DRL) to train a neural network controller. The effectiveness and practicality of the proposed method have been validated on a shallow, simply supported buckled beam. The results prove that DRL can significantly increase the safety margin and effectively mitigate buckling under high load levels without requiring extra energy. Compared with conventional model-based linear and polynomial controllers, the proposed control strategy demonstrates excellent adaptability and ease of implementation. This research aims to supplement and expand the existing understanding of DRL applications in structural control, pointing towards a promising direction for future technological advancements and real-world applications.

Keywords

Deep reinforcement learning optimal active control neural network-based control model-free control algorithm nonlinear vibration

1. Introduction

Structural control is pivotal in mitigating vibrations of civil structures (e.g., buildings and bridges) induced by external dynamic loads (such as wind and seismic loads). These vibrations may jeopardize structural serviceability and integrity and pose safety risks to the public. As architectural trends lean towards more slender structural elements, nonlinear effects, particularly geometric nonlinearities, become increasingly important (Ali and Moon, 2018). While increasing the stiffness of a structure can directly reduce displacements and mitigate nonlinear effects, it also inevitably increases structural weight and the associated costs. Achieving a balance between minimizing risk and avoiding excessive material use necessitates the development of advanced, adaptive, and scalable control strategies to manage nonlinear vibration challenges effectively.

The emergence of active control methods has marked a significant advancement in vibration control in the last few decades (Datta, 2003; Spencer and Nagarajaiah, 2003). Passive control systems are normally tuned to specific resonant frequencies and have fixed parameters, while active control systems can control multiple vibration modes simultaneously, resulting in a wide adaptive range for disturbances (Saaed et al., 2015; Soong and Costantinou, 2014). Although semi-active systems have stronger adaptability than passive systems, they are still incomparable to active systems (Casciati et al., 2012; Song and Gu, 2007). In comparison, passive systems can react to dissipate energy; semi-active systems can merely modulate energy dissipation; while active control systems can actively inject or dissipate energy (Cai and Zhu, 2022). Despite concerns related to power dependency and system failure still remain, the effectiveness of active control systems can be improved with advanced technologies (Li and Zhu, 2022).

While active control methods have achieved considerable success in linear systems, extending their applications to nonlinear systems still faces challenges. Nonlinear systems are inherently more complex and sensitive to initial conditions (Abdel-Rohman and Leipholz, 1978; Astolfi et al., 2008; Iqbal et al., 2017; Nayfeh et al., 1980; Xing et al., 2017). Direct implementation of active linear controllers to nonlinear systems relies on linearizing nonlinear systems around specific operating points (Agrawal et al., 1998). However, this approach has limitations because errors are introduced whenever deviations from these predetermined points occur (Tomasula et al., 1996). Historically, the methods for solving nonlinear feedback control problems can be classified into two main categories (Li and Todorov, 2004), based on Bellman’s optimality principle and Pontryagin’s maximum principle, respectively (Bryson and Ho, 1975; Lewis et al., 2012). While groundbreaking in their time, these conventional methods necessitate a precise understanding of the system’s mathematical model and parameters. These methods typically aim to minimize a polynomial performance index, and by incorporating higher-order performance indices based on the linear quadratic regulator (LQR) problem, a nonlinear control force can be deduced through cost function minimization, yielding a control force that becomes a nonlinear function of the state vector (Agrawal et al., 1998; Shefer and Breakwell, 1987; Suhardjo et al., 1992; Wu et al., 1995). However, these strategies encounter limitations when applied to high-dimensional systems, as the Hamilton–Jacobi–Bellman equation, central to their optimization process, becomes increasingly difficult to solve. Sliding mode control (SMC) (Mahmoudi et al., 2019) and its variant, fuzzy sliding mode control (FSMC), enrich the nonlinear control toolkit by offering robustness against external disturbances from their capacity to enforce a specific control law once the system’s state attains the sliding surface, thereby guaranteeing stability under diverse conditions. Nevertheless, they may cause chattering phenomena or increasing computational overhead (Allen et al., 2000; Yang et al., 1996; Zhao et al., 2000). Additional nonlinear control techniques include the pulse control (Pantelides and Nelson, 1995), perturbation method (Jansen, 2008), stochastic dynamic programming (Lü et al., 2020), and iterative LQR (Li and Todorov, 2004). The performance of these controllers degrades when the modeling assumptions cannot accurately capture the nonlinear dynamics.

Development in deep reinforcement learning (DRL) presents paradigm-shifting solutions for nonlinear vibration control problems. DRL-based controllers can generate optimal control strategies through direct interaction with environments without the need for predefined mathematical models (Li, 2017). Researchers have extensively utilized DRL to solve problems across various domains, including robotics, finance, and healthcare. The deep Q-network (DQN) proved the potential of DRL in discrete control problems (Mnih et al., 2013). Later, deep deterministic policy gradients (DDPG) performed well in high-dimensional, continuous action spaces (Lillicrap et al., 2015). Trust region policy optimization (Schulman et al., 2015) and its variant, proximal policy optimization (Schulman et al., 2017), demonstrated robustness and stability in complex robotic tasks. Twin delayed DDPG (TD3) further reduced overestimation bias as an improvement of DDPG (Fujimoto et al., 2018). At the same time, soft actor-critic (Haarnoja et al., 2018) optimizes the trade-off between exploration and exploitation, making it effective for complex, unknown environments. In addition, studies on hierarchical reinforcement learning (Nachum et al., 2018) and inverse reinforcement learning (Arora and Doshi, 2021) facilitated more complex decision-making processes and enabled systems to learn from human behavior, respectively (Barto and Mahadevan, 2003).

The feasibility of using RL in structural control was validated on a full-scale tensegrity structure in the earlier days (Adam and Smith, 2008). While DQN has been proven successful in generating discrete control signals in previous applications (Rahmani et al., 2019), more research has been conducted to explore the implementation of DRL under different conditions. By considering uncertainties in seismic inputs, the Q-learning RL algorithm was utilized to tune the gains of a fuzzy proportional derivative controller in order to achieve the stability of high-rise building vibration (Khalatbarisoltani et al., 2019). An actor-only RL method was used for semi-active control of a simply-supported aluminum beam (Pisarski and Jankowski, 2023). This method showed relatively low computational complexity in real-time (Panda et al., 2024). Besides, gradient descent algorithms were used in some research for active control (Nagendra et al., 2017; Eshkevari et al., 2023). Previous work by the authors (Zhang and Zhu, 2023) has also shown promising results in controlling vibrations of a linear shear-building by adopting DDPG. However, there is still a lack of research in nonlinear vibration control applications using DRL (Xie et al., 2020).

This study proposed a model-free active control strategy for nonlinear vibration control. A neural network (NN) controller is trained by DRL to eliminate uncertainties and complexities of the target nonlinear system. The proposed strategy was tested on a shallow, simply-supported buckled beam with nonlinearities up to the 3^rd order. Under certain loading conditions, the beam may lose stability at load levels below the inherent strength of the material and suffer snap-through buckling (Pinto and Gonçalves, 2000). The DRL-based methodology demonstrated superior performance through comparison with prevailing model-based polynomial controllers. The strategy not only attenuated vibrational responses but also significantly enhanced the safety margins of the structural element, effectively reducing the risk of snap-through buckling while maintaining a conservative energy consumption.

The key contributions of this research can be summarized as follows: (a) Development of an advanced active control methodology for nonlinear vibration control, which outperforms conventional model-based polynomial controllers. (b) Introduction of a highly adaptable and easily implementable active nonlinear control strategy with customizable training parameters to suit various application scenarios.

The structure of this paper is organized in the following manner: Section 2 elaborates on the DRL controller that we propose. Section 3 provides a detailed description of the nonlinear system utilized for testing. Section 4 discloses the implementation details and the simulation results under various test conditions. Lastly, Section 5 offers a summary and conclusion of the work presented.

2. Nonlinear vibration control

2.1. Classical model-based optimal control

For a fully observable and controllable dynamical system, the state-space representation can be expressed as

\dot{x} = f [x (t), u (t), t] .

(1)

where

x (t)

is the state vector,

u (t)

is the control vector,

t

states the time, and

f

is the system’s dynamic function. In the case of a linear system, optimal control approaches such as the LQR controller are directly applicable. The optimal feedback gain

K_{l q r}

is determined through the resolution of the algebraic Riccati equation (ARE). Consequently, the control action u is formulated as

u = - K_{l q r} x,

(2)

K_{l q r}

is the optimal feedback gain that minimizes the quadratic cost function

J = \int_{0}^{\infty} (x^{T} Q x + u^{T} R u) d t,

(3)

where

Q

and R are the state and control force weight matrices, respectively. The term

x^{T} Q x

and

u^{T} R u

represent the costs associated with the states and control forces.

The efficacy of linear control gains, however, diminishes when addressing nonlinear systems unless these systems are linearized around specific operational points. This limitation necessitates the exploration of nonlinear control strategies. As detailed by Suhardjo et al. (1992), a polynomial controller offers one such alternative. Recursive equations can be formulated and solved by expressing the state equation and control force as polynomial functions. Although these equations can extend to any polynomial degree, it is essential to note that as the order increases, so does the computational complexity; at the same time, the incremental contribution to control performance becomes marginal. This research compares the polynomial controller up to the 3^rd order with the DRL-based control methodology proposed in the following subsection.

2.2. DRL-based control

To address the abovementioned problems, the current study introduced DRL to train an optimal NN controller without directly solving the ARE. The control problem is systematically formalized into a DRL paradigm, wherein the system’s dynamics are discretized and encapsulated within a Markov decision process (MDP) characterized by ( S , A , P , G ). S represents the set of states, which equals the system state vector $x$ . A represents the action set, while P and G designate the transition probability distribution and reward function, respectively. In contrast to the feedback control system, the environment is equivalent to the plant model, and the agent acts as the controller. At every time step $t$ , the agent experiences the state s_t ∈ S and takes the action a_t ∈ A according to policy $μ$ . The agent receives the reward r_t ∈ G (s_t, a_t) and ends up in a new state s_t₊₁ with the probability P . The agent aims to find the optimal policy $μ^{*}$ to maximize the total discounted reward, defined as the return G_t, with the discount factor γ ∈ [0, 1]

G_{t} = r_{t + 1} + γ r_{t + 2} + {γ^{2} r}_{t + 3} + \dots = \sum_{k = 0}^{\infty} γ^{k} r_{t + k + 1} .

(4)

For the specific problem of vibration suppression under variable excitation conditions, the step reward is formulated as a negative quadratic cost

r_{t} = - s_{t}^{T} Q_{s} s_{t} - a_{t}^{T} {Ra}_{t} .

(5)

Thus, as the expected reward represents the future control performance, maximizing the total return in equation (5) is equivalent to minimizing the quadratic cost function in equation (3). In this way, the controller’s performance is optimized.

By utilizing a model-free DRL algorithm, the study leverages the benefit of not requiring any precise knowledge of system dynamics, thereby enhancing the controller’s robustness to adapt to uncertainties prevalent in real-world control applications. The illustration of training a control policy for a target nonlinear plant by DRL is shown in Figure 1.

Figure 1.

DRL structure.

In the present investigation, the system can oscillate freely or under external excitation for a predetermined duration. The control policy is subsequently updated based on the data recorded. This data-driven update mechanism utilizes the TD3 algorithm, obviating the need for solving complex nonlinear equations traditionally associated with model-based controllers. TD3 is a model-free, online, and off-policy algorithm that employs a deterministic policy for action selection. The TD3 agent comprises two key components, namely the “actor” and the “critic,” both implemented using deep neural networks (DNN). The actor is responsible for generating the control actions based on the current system state or observations. In comparison, the critic estimates the Q-value of state-action pairs, thereby guiding the policy update mechanism. TD3 represents an advanced variation of the DDPG algorithm and incorporates three key improvements to mitigate the overestimation bias in the Q-value function. The algorithm employs double Q-learning by instantiating two separate critic networks and adopting the lower of the two Q-value estimates for policy evaluation. The policy is updated at a reduced frequency compared with the Q-function updates, contributing to more stable learning dynamics. A small amount of clipped random noises is averaged among the mini-batches and added to the selected action when updating the value function to make the regularization smoother. In TD3, exploration is facilitated by incorporating noise into the actions determined by the policy network throughout the training phase, thereby enriching the agent’s experience by promoting the discovery of a broad range of strategies within the action space.

Upon completing the training process outlined in Algorithm 1, the trained actor alone will serve as an offline controller. It can generate control forces in response to observed system states, like a traditional model-based controller, but with the additional benefits of enhanced adaptability and robustness.

3. Nonlinear control problem formulation

3.1. Buckled beam

To evaluate the efficacy of the proposed DRL-based control methodology, this section investigates the nonlinear oscillations of a shallow, simply supported buckled beam subjected to transversal dynamic loads. The single-degree-of-freedom (SDOF) model derivation is aligned with the work by Pinto and Gonçalves (2002) who employed polynomial controllers up to the third order for the same problem.

Consider a homogeneous elastic simply supported beam, as shown in Figure 2. The section properties are defined by length L, Young’s modulus E, and moment of inertia I. Assuming the response $W$ and the lateral load $V$ are symmetric. For a large initial deflection, non-symmetric responses involving higher vibration modes may occur. In this study, the initial deflection is assumed to be very shallow, and only the first mode is considered. The equation of motion of the undamped SDOF model can be written as follows in a dimensionless form

\ddot{α} + (1 - p + 2 β^{2}) α + 3 β α^{2} + α^{3} = - v,

(6)

w (x, t) = α (t) \sin (π x),

(7)

p = \frac{P}{P_{c r}} = \frac{P L^{2}}{π^{2} E I}, v = \frac{V L^{4}}{2 π^{4} E I r}, w = \frac{W}{2 r}, x = \frac{X}{L},

(8)

β = \pm \frac{L}{6 π r} \sqrt{- 24 + 18 p \pm 6 \sqrt{9 p^{2} - 8}} .

(9)

where

α

represents the amplitude of the first mode.

p

v

w

x

are dimensionless quantities. The initial axial load of the beam is P, P_cr is the static Euler critical load of the beam subjected to axial compressive load, and r is the radius of gyration. Amplitude

β

is the solution to the static buckling problem, assuming the initial static deflection form is the first buckling mode.

Figure 2.

Buckled beam setup.

The control mechanism of the buckled beam consists of applying moment couples at specific points along the beam axis. The dimensionless system equation of motion of the controlled beam is

\ddot{α} + (1 - p + 2 β^{2}) α + 3 β α^{2} + α^{3} - \sum_{i = 1}^{k} 2 \cos (π d_{i}) u_{i} = - v,

(10)

where u_i is the dimensionless quantity of the ith control moment located at a distance d_i from the origin, and k is the number of the applied control moment.

3.2. Static behavior

The behavior of the buckled beam under static loading is initially analyzed. Considering a time-independent transversal load v, the static equilibrium equation could be obtained from equation (6) as

(1 - p + 2 β^{2}) α + 3 β α^{2} + α^{3} = - v .

(11)

For p = 1.005 and $β$ = 1, the static response of the buckled beam with a very shallow arch is shown in Figure 3, indicating two stable equilibrium positions ( $α$ = −2.005 and $α$ = 0) and one unstable equilibrium in the middle (the black cross markers). The snap-through buckling path (red dash line) from point P1 to P2 occurs upon a slight incremental load from the local peak at P1, the deflection significantly increases. Similar behavior is observed from P3 to P4 if the load is reduced from P3. This dynamic behavior leads to oscillations around point P2 or P4 in the absence of damping.

Figure 3.

Snap-through behavior of the bulked beam.

3.3. Dynamic behavior

If a linear viscous damping equivalent to 3.54% of critical damping of the linearized system is assumed, and two actuators are located at L/3 and 2L/3, the equation of motion for the uncontrolled system becomes

\ddot{α} + 0.1 \dot{α} + 1.995 α + 3 α^{2} + α^{3} - 2 u = - v .

(12)

In the uncontrolled scenario, the system’s final state under free vibration is sensitive to its initial condition, as two stable equilibrium points exist in the system. Figure 4 illustrates the displacement responses of the beam under free vibration for two distinct initial states, where the escape point is

α

= −1.00. The MATLAB function ode45 was employed to perform the integration for the numerical solution of these nonlinear equations.

Figure 4.

Free vibration response of the uncontrolled system.

With the weighting matrix in equation (3) chosen to be $Q (1, 1) = 0.1, Q (2, 2) = 0.1$ and R = 1 (other coefficients are set to zero), the 1^st order LQR controller was implemented, and its impact on the free vibration behavior near the escape point is depicted in Figure 5(a). The escape point shifted to $α$ = −1.11.

Figure 5.

Free vibration response of the system with (a) the first order controller; (b) the second and third order controllers.

By employing higher-order controllers based on the polynomial strategies proposed by Suhardjo et al. (1992), the following observations were made. For the 2^nd order control, the system invariably settles to the first equilibrium point $α$ = 0, regardless of where the initial position is, even at the second equilibrium point $α$ = −2.00. A similar trend is seen with the 3^rd order control. Figure 5(b) shows the displacement response from an initial state $α$ = −2.00 with the 2^nd order control and changing the initial state to $α$ = −3.00 with the 3^rd order control. The weighting coefficients for the higher-order terms of the polynomial controller were set to zero for this analysis.

When subjected to a sudden step load, the system has the possibility to cross the energy barrier, resulting in dynamic buckling even at loads lower than the static critical load. The load needed for this dynamic transition increases with the order of the control; however, the increase becomes marginal beyond third-order control. In the case of an uncontrolled system, the escape load stands at 0.31, which is significantly lower (by 18.4%) than the static critical load of 0.38.

Figure 6 shows the response time history of the beam subject to a step load of infinite duration (v = 0.42) under the regulation of the active controllers of varying orders. At this load level, the 1^st order controller proves insufficient in averting snap-through buckling. The beam transitions to a post-buckling state, indicative of the limitations of linear control schemes for systems exhibiting nonlinear behaviors. In contrast, the 2^nd and 3^rd order controllers are competent in retaining the system within the first potential well, thereby preventing buckling. In the following section, the performance of the DRL-trained controller is presented and compared to the model-based polynomial controllers.

Figure 6.

Time response of different orders of control under a step load of infinite duration (v = 0.42).

4. Results and discussions

4.1. DRL training

The subsequent section will delve into the performance and characteristics of the controller trained using DRL. Based on the behavior of the system mentioned above, the DRL training is performed by applying a step load (0.48) of infinite duration slightly higher than the escape load of the 3^rd order (0.47) controller. The training allows the agents to explore during the interaction with the environment. Each training episode lasts for 25 s. The training methodology employs the TD3 algorithm, featuring a training duration of 800 episodes. The training was conducted using MATLAB on an Intel i7 9700 CPU. From the onset, exploration is initiated by implementing a noise-added action policy, with the exploration rate starting at an initial action noise variance of 0.001 and a decay rate of $1 \times 10^{- 5}$ . The batch size for training is set at 128. The architecture for the actor-network includes two hidden layers with 16 nodes for each layer, and employs a hyperbolic tangent (Tanh) activation function. The actor learning rate is set at $1 \times 10^{- 4}$ . Both critic networks feature three hidden layers with 64 nodes in each layer, with rectified linear units (ReLU) as the activation function and a learning rate of 0.001. The discount factor is 0.99.

The observation includes the system state ${[α, \dot{α}]}^{T} .$ The reward is the negative quadratic cost (equation (5)) by setting the weighting matrix the same as the 1^st order controller, that is, $Q (1, 1) = 0.1, Q (2, 2) = 0.1$ and R = 1 (other coefficients are set to zero). The performance cost $J$ is the combination of state response and control force cost

J = \sum_{i = 1}^{i = \max s t e p} ([α_{i}, {\dot{α}}_{i}] Q {[α_{i}, {\dot{α}}_{i}]}^{T} + u_{i} R u_{i}) .

(13)

During the training process, the DRL-trained controller’s performance was analyzed with an initial system state at $α$ = −1.50. As depicted in Figure 7, the TD3 agent rapidly outperforms all three traditional controllers, highlighting its advanced adaptive capabilities. The standard deviation of over 5 trials shows a smooth training process. Critically, the optimal quadratic performance cost J of the TD3 agent is found to be 3.51 under the step load of 0.48. This marks a dramatic reduction of approximately 90%, compared with the 3^rd order controller, which registered a $J$ value of 35.26. Notably, the 3^rd order controller was unable to avert snap-through buckling at this elevated load level. The substantial disparity in performance $J$ between the DRL-trained and the 3^rd order controllers underscore the former’s superior capability in managing complex nonlinear dynamics, particularly under challenging loading conditions. The TD3 agent avoids snap-through buckling and accomplishes this with significantly less energy consumption, demonstrating its robustness and efficiency.

Figure 7.

The evolution of the cost function $J$ with the iteration of DRL training.

Notably, although the DRL training is done under a single step load (v = 0.48), the same trained DRL-based controller will be tested under different loading conditions in this section, including step loads with different amplitudes and random excitations.

4.2. Escape point

The DRL-trained controller’s performance is further assessed by monitoring the system’s free vibration response. Results obtained using the same weighting matrix are visualized in Figure 8. The initial displacement is set at $α$ = −1.50. The displacement response of the DRL-trained controller declines more swiftly than that of the 3^rd order polynomial controller. In addition, the DRL-trained agent exhibits the lowest performance cost $J$ , registering a value of 6.96, compared with a cost of 7.67 for the 3^rd order polynomial controller. Like high-order polynomial controllers, the DRL-trained agent returns to the zero-equilibrium state irrespective of the initial conditions.

Figure 8.

Time response of TD3 agent controlled free vibration of the buckled beam.

4.3. Safety region

The system’s displacement responses under the effect of different controllers are computed and compared under the step load v = 0.45 in Figure 9. After about 3 s, the 1^st and 2^nd order controllers could not keep the beam in the pre-buckling position, while the displacement of the TD3 agent controller remains in the safe region and converges quickly. Increasing the step load greater than the escape load will lead the system to experience dynamic buckling. The variation of escape load v_e is detailed in Table 1, where Δv_e is the improvement in comparison with the uncontrolled case. Apparently, the trained agent increased the escape load of the system considerably by 22.58%, compared to the 3^rd order polynomial controller.

Figure 9.

Response time histories of the controlled buckled beam system under a step load.

Table 1.

Variation of the escape load.

Controller	No control	Traditional polynomial controller			DRL (TD3)
Controller	No control	1^st	2^nd	3^rd	DRL (TD3)
v_e	0.31	0.39	0.43	0.47	0.54
Δv_e (%)		25.81	38.71	51.61	74.19

Figures 10(a) and (b) display the control force history and the force-displacement relationship under the chosen step load. For the 1^st and 2^nd order controllers, the control force rapidly escalates after buckling, while the DRL-based controller requires slightly more peak force than the 3^rd order controller but achieves significantly better performance in terms of the cost function J. The DRL-trained agent tends to generate a larger control force at lower displacement levels, resulting in a smaller resting displacement. This aspect is readily observable from the force-displacement plot in Figure 10(b). The peak and root-mean-square (RMS) values of displacement, velocity, and force for different controllers are compared in Table 2. In terms of system safety and robustness, the DRL-trained controller significantly outperforms the traditional controllers. Its capability to maintain the system within a safe region under varied loading conditions, along with its ability to do so more efficiently (in terms of control force applied and energy used), underscores its suitability for complex nonlinear systems.

Figure 10.

Response of the controlled buckled beam under step load v = 0.45 (a) Control force history; (b) Control force-displacement relationship.

Table 2.

Responses and control forces of the buckled beam system under step load v = 0.45 with different controllers.

Controller		Displacement		Velocity		Control force		Cost $J$
Controller		Peak	RMS	Peak	RMS	Peak	RMS	Cost $J$
Traditional	1^st order	2.45	1.87	0.82	0.25	0.24	0.11	92.53
	2^nd order	2.08	1.60	0.47	0.16	0.45	0.30	84.81
	3^rd order	0.55	0.42	0.26	0.07	0.09	0.05	4.85
DRL	TD3	0.39	0.27	0.22	0.05	0.11	0.06	2.76

4.4. Performance cost

An important measure of the controller’s performance is the performance cost $J$ under different levels of step loads, as a weighted combination of control effort and system states response reduction. The performance function $J$ of the system under varying levels of step loads was recorded and plotted in Figure 11. For loads below the escape load (v < 0.24), the TD3 agent exhibits slightly higher $J$ compared to traditional polynomial controllers. As the system surpasses its escape load, a sharp uptick in $J$ value is observed for all controllers. However, the rate of increase for the TD3 agent is more moderate, which could be attributed to its higher escape load. The behavior of the TD3 agent at lower loads might be an artifact of its training regimen. The observed phenomenon may result from the training conditions, specifically the choice of step load levels, which were geared towards preventing post-buckling behavior rather than optimizing performance cost at lower load levels. The training was focused on preventing buckling under loads greater than the escape load of the 3^rdorder controller. Consequently, the TD3 agent may not have been optimized for minimizing the performance function under conditions where the system is naturally stable.

Figure 11.

Performance cost $J$ vs step load $v$ for different controllers.

4.5. Random excitation

To evaluate the efficacy of the controller trained through DRL under uncertain conditions, an imposed random excitation lasting 1000 s was applied to the system. The displacement response under random white-noise excitation, as illustrated in Figure 12, makes it evident that neither the linear controller nor the 2^nd order controller could prevent the beam from experiencing snap-through buckling. In contrast, both the 3^rd order polynomial controller and the TD3 agent effectively mitigated the risk of buckling under the applied level of random excitations. This performance attests to a notable robustness in both controllers when dealing with unpredictable external influences.

Figure 12.

Time history of random excitation.

Evaluation metrics, nondimensionalized with the responses of an uncontrolled scenario, were selected to analyze the maximum and RMS states of the controlled system. As indicated in Table 3, the values of these criteria decrease as the control order ascends. This decreasing trend suggests that increasing nonlinearity in the controllers corresponds to enhanced control performance. Interestingly, the TD3 agent registered the lowest values among all the tested controllers, suggesting that the agent may have learned some intrinsic features of the nonlinear system. One salient observation pertains to the TD3 agent’s superior capacity to minimize maximum displacement, even when compared to the 3^rd order controller. Furthermore, this was achieved while necessitating a substantially lower maximum control force, 0.45 for the TD3 agent versus 0.69 for the 3^rd order controller. This underscores the DRL-trained controller’s heightened efficiency and effectiveness.

Table 3.

Evaluation criteria for different controllers under random excitation.

Response		Traditional polynomial controller			DRL (TD3)
Response		1^st order	2^nd order	3^rd order	DRL (TD3)
Displacement	Max	0.75	0.56	0.32	0.31
Displacement	RMS	0.47	0.22	0.17	0.12
Velocity	Max	0.39	0.38	0.36	0.35
Velocity	RMS	0.36	0.36	0.35	0.31

Increasing the order of a polynomial controller can indeed enhance control performance, but this improvement is associated with significantly escalated computational demands as the controller’s complexity rises. In summary, the proposed DRL control strategy outperforms traditional model-based control methods in terms of adaptability and efficiency. Since DRL training can balance the trade-off between control performance and operation costs, the proposed control strategy shows excellent flexibility. For instance, a higher escape load and broader safety margin can be achieved by setting a larger step load into the training process. In addition, the DRL method learns from the data directly without requiring any model information. This model-free approach can avoid the problems associated with system uncertainties and model inaccuracies, leading to robust control performance and great adaptability in broad vibration scenarios.

5. Conclusions

This study proposed an innovative control strategy to realize model-free nonlinear active vibration control by using DRL, which was proved more flexible and adaptable than the traditional model-based polynomial control strategies.

An NN controller was trained by the TD3 algorithm to realize optimal control performance. The tests were conducted on a simply supported buckled beam in various vibration scenarios. Superior performances of the TD3-trained NN controller were demonstrated under both static loadings and random excitations. Compared with the system controlled by the 3^rd order polynomial controller, the DRL-controlled system improved the safety margin and the escaped load was increased by nearly 20%. Moreover, such peformance improvement did not increase the control force correspondingly, making it more energy efficient than the conventional controllers. Although being trained under step loads, the NN controller could reduce displacement and control force more effectively than the 3^rd order polynomial controller under random excitations. This reveals the robustness and adaptability of the proposed method.

Even though DRL-based methods can offer robust and scalable solutions for nonlinear vibration control, real-world applications involve more complex characteristics, including uncertainty, noise, and operational variability of the control target. How to train agents to adapt practical vibration scenarios effectively remains a question. Additional training and restructuring of NN is required in future research to enhance the robustness of the controller and solve problems of uncertainties.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Research Grants Council of Hong Kong (Project Nos. CRS_PolyU503/23, 15214620, PolyU R5006-23), and the Hong Kong Branch of the National Rail Transit Electrification and Automation Engineering Technology Research Center (K-BBY1).

ORCID iDs

Yi-Ang Zhang

Songye Zhu

References

Abdel-Rohman

Leipholz

(1978) Active control of flexible structures. Journal of the Structural Division 104(8): 1251–1266.

Adam

Smith

(2008) Reinforcement learning for structural control. Journal of Computing in Civil Engineering 22(2): 133–139.

Agrawal

Yang

(1998) Non-linear control strategies for Duffing systems. International Journal of Non-Linear Mechanics 33(5): 829–841.

Ali

Moon

(2018) Advances in structural systems for tall buildings: emerging developments for contemporary urban giants. Buildings 8(8): 104.

Allen

Bernelli-Zazzera

Scattolini

(2000) Sliding mode control of a large flexible space structure. Control Engineering Practice 8(8): 861–871.

Arora

Doshi

(2021) A survey of inverse reinforcement learning: challenges, methods and progress. Artificial Intelligence 297: 103500.

Astolfi

Karagiannis

Ortega

(2008) Nonlinear and Adaptive Control with Applications. Berlin: Springer.

Barto

Mahadevan

(2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 13(1–2): 41–77.

Bryson

Y-C

(1975) Applied Optimal Control: Optimization, Estimation, and Control. London: Routledge.

10.

Cai

Zhu

(2022) The nexus between vibration-based energy harvesting and structural vibration control: a comprehensive review. Renewable and Sustainable Energy Reviews 155: 111920.

11.

Casciati

Rodellar

Yildirim

(2012) Active and semi-active control of structures - theory and applications: a review of recent advances. Journal of Intelligent Material Systems and Structures 23(11): 1181–1195.

12.

Datta

(2003) A state-of-the-art review on active control of structures. ISET Journal of Earthquake Technology 40(1): 1–17.

13.

Fujimoto

Hoof

Meger

(2018) Addressing function approximation error in actor-critic methods. In: International conference on machine learning, Stockholm, Sweden, pp. 1587–1596. PMLR.

14.

Haarnoja

Zhou

Abbeel

et al. (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, Stockholm, Sweden, pp. 1861–1870. PMLR.

15.

Iqbal

Ullah

Khan

, et al. (2017) Nonlinear control systems-A brief overview of historical and recent advances. Nonlinear Engineering 6(4): 301–312.

16.

Jansen

(2008) A perturbation method for nonlinear vibrations of imperfect structures: application to cylindrical shell vibrations. International Journal of Solids and Structures 45(3–4): 1124–1145.

17.

Khalatbarisoltani

Soleymani

Khodadadi

(2019) Online control of an active seismic system via reinforcement learning. Structural Control and Health Monitoring 26(3): e2298.

18.

Lewis

Vrabie

Syrmos

(2012) Optimal Control. Hoboken, NJ: John Wiley & Sons.

19.

(2017) Deep reinforcement learning: an overview. ArXiv preprint arXiv:1701.07274.

20.

Todorov

(2004) Iterative linear quadratic regulator design for nonlinear biological movement systems. ICINCO 1: 222–229.

21.

J-Y

Zhu

(2022) Self-powered active vibration control: concept, modeling, and testing. Engineering 11: 126–137.

22.

Lillicrap

Hunt

Pritzel

, et al. (2015) Continuous control with deep reinforcement learning. ArXiv preprint arXiv:1509.02971.

23.

Lü

Wang

, et al. (2020) Nonlinear stochastic optimal control using piezoelectric stack inertial actuator. Shock and Vibration 2020: 1–7.

24.

Mahmoudi

Ghaffarzadeh

Ahani

, et al. (2019) Sliding mode control of linear structures and a Duffing system using active tendons. Proceedings of the Institution of Civil Engineers - Engineering and Computational Mechanics 172(3): 106–117.

25.

Mnih

Kavukcuoglu

Silver

, et al. (2013) Playing atari with deep reinforcement learning. ArXiv preprint arXiv:1312.5602.

26.

Nachum

Lee

, et al. (2018) Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems 31.

27.

Nagendra

Podila

Ugarakhod

, et al. (2017) Comparison of reinforcement learning algorithms applied to the cart-pole problem. 2017 international conference on advances in computing, communications and informatics (ICACCI), Manipal. India: IEEE, 26–32.

28.

Nayfeh

Mook

Holmes

(1980) Nonlinear oscillations.

29.

Panda

Chopra

Matsagar

, et al. (2024) An iterative gradient descent-based reinforcement learning policy for active control of structural vibrations. Computers & Structures 290: 107183.

30.

Pantelides

Nelson

(1995) Continuous pulse control of nonlinear structures. Computers & Structures 55(6): 997–1006.

31.

Pinto

Gonçalves

(2000) Non-linear control of buckled beams under step loading. Mechanical Systems and Signal Processing 14(6): 967–985.

32.

Pinto

Gonçalves

(2002) Active non-linear control of buckling and vibrations of a flexible buckled beam. Chaos, Solitons & Fractals 14(2): 227–239.

33.

Pisarski

Jankowski

(2023) Reinforcement learning‐based control to suppress the transient vibration of semi‐active structures subjected to unknown harmonic excitation. Computer-Aided Civil and Infrastructure Engineering 38(12): 1605–1621.

34.

Rahmani

Chase

Wiering

, et al. (2019) A framework for brain learning-based control of smart structures. Advanced Engineering Informatics 42: 100986.

35.

Saaed

Nikolakopoulos

Jonasson

, et al. (2015) A state-of-the-art review of structural control systems. Journal of Vibration and Control 21(5): 919–937.

36.

Eshkevari

Sadeghi Eshkevari

Sen

, et al. (2023) Active structural control framework using policy-gradient reinforcement learning. Engineering Structures 274: 115122.

37.

Schulman

Levine

Abbeel

, et al. (2015) Trust region policy optimization. International conference on machine learning. Lille, France: JMLR, 1889–1897.

38.

Schulman

Wolski

Dhariwal

, et al. (2017) Proximal policy optimization algorithms. ArXiv preprint arXiv:1707.06347.

39.

Shefer

Breakwell

(1987) Estimation and control with cubic nonlinearities. Journal of Optimization Theory and Applications 53(1): 1–7.

40.

Song

(2007) Active vibration suppression of a smart flexible beam using a sliding mode based controller. Journal of Vibration and Control 13(8): 1095–1107.

41.

Soong

Costantinou

(2014) Passive and Active Structural Vibration Control in Civil Engineering. Berlin: Springer.

42.

Spencer

Jr Nagarajaiah

(2003) State of the art of structural control. Journal of structural engineering 129(7): 845–856.

43.

Suhardjo

Spencer

Jr Sain

(1992) Non-linear optimal control of a Duffing system. International Journal of Non-Linear Mechanics 27(2): 157–172.

44.

Tomasula

Spencer

Jr Sain

(1996) Nonlinear control strategies for limiting dynamic response extremes. Journal of Engineering Mechanics 122(3): 218–229.

45.

Lin

Soong

(1995) Non-linear feedback control for improved peak response reduction. Smart Materials and Structures 4(1A): A140–A148.

46.

Xie

Ebad Sichani

Padgett

, et al. (2020) The promise of implementing machine learning in earthquake engineering: a state-of-the-art review. Earthquake Spectra 36(4): 1769–1801.

47.

Xing

Wen

Liu

, et al. (2017) Event-triggered adaptive control for a class of uncertain nonlinear systems. IEEE Transactions on Automatic Control 62(4): 2071–2076.

48.

Yang

(1996) Control of seismic-excited buildings using active variable stiffness systems. Engineering Structures 18(8): 589–596.

49.

Zhang

Y-A

Zhu

(2023) Novel model-free optimal active vibration control strategy based on deep reinforcement learning. Structural Control and Health Monitoring 2023: 1–15.

50.

Zhao

, et al. (2000) Sliding mode control of buildings with base‐isolation hybrid protective system. Earthquake engineering & structural dynamics 29(3): 315–326.