Actor–critic-based decision-making method for the artificial intelligence commander in tactical wargames

Abstract

In a tactical wargame, the decisions of the artificial intelligence (AI) commander are critical to the final combat result. Due to the existence of fog-of-war, AI commanders are faced with unknown and invisible information on the battlefield and lack of understanding of the situation, and it is difficult to make appropriate tactical strategies. The traditional knowledge rule-based decision-making method lacks flexibility and autonomy. How to make flexible and autonomous decision-making when facing complex battlefield situations is a difficult problem. This paper aims to solve the decision-making problem of the AI commander by using the deep reinforcement learning (DRL) method. We develop a tactical wargame as the research environment, which contains built-in script AI and supports the machine–machine combat mode. On this basis, an end-to-end actor–critic framework for commander decision making based on the convolutional neural network is designed to represent the battlefield situation and the reinforcement learning method is used to try different tactical strategies. Finally, we carry out a combat experiment between a DRL-based agent and a rule-based agent in a jungle terrain scenario. The result shows that the AI commander who adopts the actor–critic method successfully learns how to get a higher score in the tactical wargame, and the DRL-based agent has a higher winning ratio than the rule-based agent.

Keywords

Actor–critic artificial intelligence agent deep reinforcement learning decision-making

1. Introduction

At present, the development of artificial intelligence (AI) has led to significant breakthroughs in the decision-making area. Research on deep learning (DL) and reinforcement learning (RL) has led to important progress in computer vision,^1,2 the game of Go,³ robot control,^4,5 and many other domains. Some researchers have attempted to solve complex reinforcement learning problems in the real-time strategy (RTS) game. The combination of uncertain conditions, incomplete information, and multi-military actions adds complexity for AI commanders to make appropriate decisions. Correspondingly, this also poses a great challenge to deep reinforcement learning (DRL).

Tactical wargames combine elements of both simulation and gaming in a military environment. Like the RTS game, they also have different factions, each with its own set of units and abilities. During the game, a player must perform many tactical actions, such as moving, scouting, attacking, and occupying an opponent’s base, with the goal of obtaining a higher score. Each tactical action has different parameters. For example, an attack has stop firing and mobile firing, with different weapons to choose from.

In the area of tactical wargame research, AI commander decision-making is among the most challenging area, because it requires mastery of battlefield information, implementation of overall strategic planning, and execution of micro-actions. To defeat the human players, AI commander modeling requires learning about human thought process. The decision-making process of AI commanders is similar to that of humans.⁶ The process of the commander’s decision-making is shown in Figure 1.

First of all, defining the mission includes mission analysis where the action rule guidance and goals are considered, and describes the essential tasks to be accomplished and the purpose to be achieved. Then one gathers the information, and starts situation assessment to assess the impact of both forces and the battlefield environment. On this basis, the different tactical courses of action (COAs) are analyzed to outline the required forces, force deployment, action time, and time to achieve the target, and reserve an action plan to choose the best COA and implement it.

RL is a way to solve the decision-making problem, but its results are severely affected when faced with huge state space and action space. DRL, combining the advantage of the perception of DL and control of RL, can greatly improve the practicality of decision problem solving. The purpose of this study is to apply the DRL method to the decision-making of AI commanders. This paper is organized as follows. In Section 2, a literature review on the application of the DRL approach in image recognition, especially in the game field, is presented. Section 3 conducts a DRL research environment for AI commander decision-making, which integrates the tactical wargame and machine learning algorithm lab, such as in TensorFlow, making it easier to try different AI algorithms to train the AI agent. In Section 4, a dedicated DRL agent model is proposed and the running process of the actor–critic (AC) model is introduced. The DRL agent and rule-based AI agent are analyzed through the combat experiments, and the effectiveness of the AC approach for commander decision-making in tactical wargames is verified in Section 5, followed by the conclusions and future study directions in Section 6.

2. Related works

The AI commander must gather the information, judge the situation, and make tactical plans to achieve the combat intention. The decision-making is a continuous dynamic process related to the change of environment and army, which has certain difficulties and complexities. In order to solve decision-making problems, scholars have put forward some effective methods and models. These methods include the decision tree, rule-based reasoning,⁷ case-based reasoning (CBR),⁸ context-based reasoning (CxBR),⁹ the finite state machine, the Bayes network, the belief–desire–intention (BDI) cognitive model,^10,11 DRL, and so on.

Developers of AI commanders for tactical wargames also use finite state machines, decision trees, and other methods that are regarded as knowledge-driven approaches. Using game theory¹² to make decisions, many human factors are involved in the selection of schemes and allocation of parameter weights. Such AI commanders cost game developers a great deal of resources to maintain and they often face a knowledge bottleneck and lack of flexibility. Once human players discover the AI commander’s habits or rules, it can be easily defeated. Unlike the knowledge-driven methods, the DRL approach runs in a data-driven manner. By using DRL techniques, we hope to address these issues and improve the flexibility and adaptability of AI commanders.

2.1. DRL method

There are some aspects of using DRL in games that we summarize here. Lecun et al.¹³ summarized the basic principles and core advantages of DL. The deep convolutional neural network (CNN) can successfully learn the feature of an image, while Alexnet¹⁴ and very deep convolutional networks (VGGs)¹⁵ use different CNN structures to reduce the error rate in object recognition. The region-based convolutional network (RCNN)¹⁶ and single shot multi-box detector (SSD)¹⁷ greatly enhance the applicability of object detection. Wade¹⁸ created surrogate models for an air and missile defense simulation using neural networks, while Amato and Shani¹⁹ learned to switch advanced strategies for a single AI agent in the civilization IV game using the RL method. Glavin et al.²⁰ proposed a RL shooting mechanism, enabling the bot to take adaptive shoots over time based on dynamic reward signals. Hosu and Rebedea²¹ combined CNNs with the Q learning algorithm of traditional RL,²² and proposed the Deep Q-Network (DQN) model for processing visual perception-based control tasks. The DQN takes the game screen information as inputs, and the game score as the reinforcement learning signal, which has been proven to be effective in Atari video games. Peng et al.²³ proposed a continual match-based training (COMBAT) framework for training advantage-actor–critic (A2C) agents in Pommerman, which is a partially observable multi-agent game environment. Oh J et al.² combined the DRL method with the Monte Carlo Tree Search (MCTS), and proposed the AlphaGo game algorithm, which adopted the DL method to grasp the “intuitive” sense of chess, and realize self-learning through reinforcement learning. Wang et al.²⁴ proposed a Belief-state Monte Carlo Tree Search (BS-MCTS) and applied it to the Phantom Go game. Liskowski et al.²⁵ tried several CNN architectures and board encodings to win the Othello game. Takada et al.²⁶ proposed a reinforcement learning algorithm to create value and policy functions for a Hex board game. These studies provide a reference for the application of DL methods in the field of wargames; in particular, this paper describes the battlefield situation quantitatively with reference to the CNN, and designs a decision-making algorithm with reference to A2C. However, it needs to be improved and adjusted according to specific applications, such as wargames.

2.2. Decision-making learning environment

Tactical wargames provide the best experimental platform for AI commanders to make decisions. Firstly, in order to win the game, the player must have a comprehensive understanding of the game situation and take action step by step. Secondly, they need to have a clear definition of victory, such as a score-based system or destroying an opponent’s base. Thirdly, wargame status data can be formalized to facilitate deep neural network learning. Fourthly, there are many different units and ability options, creating the combination of an explosive number of choices, plans, and possible game states. DeepMind and Blizzard have jointly released the StarCraft II AI research environment SC2LE,²⁷ which provides AI developers with a basic platform for intelligent decision-making. Developed on the basis of this environment, AlphaStar²⁸ defeated professional StarCraft players, and was trained directly from raw game data through supervised learning and reinforcement learning. FAIR released an extensive, lightweight, and flexible platform (ELF)²⁹ for game research, in particular for RTS games. This paper studies the decision-making of AI commanders in tactical wargames and develops an AI learning platform called ArmorCombat, which is described in the following sections.

3. Tactical wargame environment

3.1. The AI commander learning environment

We develop a tactical wargame written in Python, in which the wargame uses the popular pyGame engine for the graphical user interface (GUI), and develop all the essential components, such as a scenario map representing various terrain scenes, chess pieces representing the combat units that actually participate in the battle, and the ruling method designed according to the actual situation and combined with the probability principle. The built-in scripted AI is not intelligent enough for AI bot developers to conduct decision-making research. So, we extend the wargame into an AI research environment called ArmorCombat, which is integrated with TensorFlow for DRL algorithm usage. We try different algorithms for developing AI commanders to win the wargame. The structure of the ArmorCombat DRL environment is shown in Figure 2.

The tactical wargame shows battle scenarios and the battle status. A player enters different battle commands to play a battle game. The wargame provides a basic combat simulation environment and a set of APIs (application program interfaces); on this basis, an intelligent commander agent is developed that can make independent decisions and send action instructions to combat units. Unlike past research in the literature,^30,31 which requires a multi-agent to reach a consensus, this paper uses the commander agent to set the actions of subordinate combat units without a communication framework. The combat units try different tactical actions to get the environment feedbacks.

The AI agent module is a critical part of ArmorCombat platform, and it extracts the battlefield situation information, including the environment information and the combat unit information, from the tactical wargame. Then the feature map of the battlefield state is obtained by the CNN, and the parametric representation of tactical actions (maneuver, attack, etc.) facilitates deep neural network training. On the basis, through the AC framework for reinforcement learning, the decision instructions output into the tactical wargame from the simulation platform to obtain the combat data and the feedback value of tactical action. Under the framework of reinforcement learning, AI agents are constantly training to find a better tactical strategy.

3.2. Tactical wargame

Tactical wargames are based on the combat action of the armored unit, which we call ArmorCombat. ArmorCombat represents the game world via a hexagon map. One advantage of using hexagons is that the distance between the centers of the two cells is the same and the distance from the center to the edge of the point should be as small as possible, which is basically consistent with the characteristics of troops moving from a point to all directions. The main interface of the tactical wargame is shown in Figure 3. Each hexagon has a different color, where the color represents a different elevation; the darker the color, the higher the elevation. The numbers below the hexagonal grid are the elevation of the terrain, which is divided into multiples of 20 meters. Hexagons also have different types of terrain, such as open ground, jungle ground, rivers, etc.

ArmorCombat models all unit costs and abilities of combat units. It divides them into two camps, each with combat units such as tanks, armored vehicles, and infantry, and different units using different weapons to attack opponents. ArmorCombat initially deals with three types of units. Tanks are the most powerful units with the highest offensive capabilities that can shoot in motion. The infantry has the lowest mobility and has little offensive or defensive value, but its rocket weapons can destroy armored vehicles and tanks over long distances. Armored vehicles are more powerful than infantry, and can carry infantry to designated positions. Armored vehicles and infantry can only shoot in a stationary state. Combat units have the ability to scout, move, shoot, and occupy. They consume mobility value when passing through different kinds of terrain hexagons, and the mobility value consumed is 1, 2, 3, respectively, when passing through open ground, jungle land, and residential areas. The mobility and attack ability value of combat units are limited. When their maneuverability is consumed, they cannot continue to move until their mobility is restored for some time. The attack ability is handled in the same way. The usual A* pathfinding algorithm is used to determine the maneuvering route of the combat unit. The A* algorithm always finds the best possible path between two locations in a reasonably short time.

ArmorCombat is a turn-based tactical wargame. Generally, it takes five rounds to complete; each round is divided into five stages. According to the process rule, the red camp and the blue camp alternately perform two maneuvers and two shots in each stage. The game has a set of deduction rules for maintaining game balance. It involves maneuver rules, getting on and off rules, hidden rules, observation rules, shooting rules, same-position fighting rules, and adjudication rules at different stages.

AI commanders are divided into DRL-based AI and rule-based AI, which can fight against each other and against human players, forming several combat modes. As shown in Figure 3, ArmorCombat has a primary control point and a secondary control point, and the AI commander can control the tanks, armored vehicles, and infantry teams to occupy them and get their respective occupation scores. It can also obtain scores by killing or destroying the opposing combat unit, and at the end of the game, the camp with the highest score wins. In this study, our goal is to allow AI commanders to explore different tactics and learn how to win games.

4. Artificial intelligence commander agent based on deep reinforcement learning

Hereunder is the implementation framework for the DRL agent. On this basis, we develop an AI commander agent and try to defeat the pre-scripted agent in the tactical game.

4.1. Problem definition

In the tactical wargame, the commander agent starts with a random strategy (action list) for exploration, which can also be thought of a Markov process. In the exploration process, we can get a series of state (s), actions (a), and feedback (r), which is represented as follows:

{s_{1}, a_{1}, r_{1}, s_{2}, a_{2}, r_{2}, \dots, a_{t - 1}, r_{t - 1}, \dots}

$r_{t + 1}$ is the instantaneous reward from $s_{t}$ to $s_{t + 1}$ , so at the t moment, the total feedback can be calculated as follows:

G_{t} = r_{t + 1} + γ r_{t + 2} + γ^{2} r_{t + 3} + \dots = \sum_{k = 0}^{n} γ^{k} r_{t + k + 1}

(1)

As γ is a discount factor, $G_{t}$ is the feedback, which is more important in the current state and decreases with time.

The AI agent can utilize reinforcement learning method to continue looking for the optimal strategy and get the maximum feedback, while the AC is the classical reinforcement learning method. Next, the combat situation (s), actions (a), and rewards (r) are illustrated in the tactical wargame context.

4.1.1 Combat situation

To win the battle, the AI commander needs to master as much information as possible, such as environmental information, their own information, and enemy information. On this basis, the AI agent makes decisions and executes actions.

The main environmental information comes from the hexagonal map rendered by $N \times M$ grids, where the map size of the jungle terrain scenario is 128×128. It can represent specific information, such as terrain types, height maps, visibility, and control points.

Environment information is static, while unit information is always dynamic, which is characterized by uncertainty and incompleteness, which is of great significance for decision-making. It can be expressed as follows:

U = {i, t, p, h, m, a}

where i is the unique identifier of the combat unit used to classify units during gameplay; t is the unit type to assign the unit: tanks, armored vehicles, cannon, or infantry; p represents the unit’s position: $(x, y)$ coordinates of each unit on the map; h: is the life value where, initially, the value is set to the maximum, which is different for different unit types; m is the mobility where, initially, the value is set to the maximum, and it decreases after going through the hexagonal grid; according to the built-in rules, units can recover the mobility value after a certain round; and a is attack power where, initially, the value is set to the maximum, which will decrease after it has fired. Units can recover the attack power value after a certain round.

This combat situation information will be quantified as input to facilitate deep neural network training.

4.1.2. Action space

In ArmorCombat, there are many types of combat units, and units perform a variety of actions. For example, the action of a tank unit includes unit selecting, scouting, moving, firing, etc. The AI agent executes a sequence of multiple combat actions, interacting with the battle environment and engaging the opponent’s forces, all of which affect the final combat result. According to the state of the damage and the control point occupying score, the system gives each side a corresponding score.

Generally speaking, a strategy is composed of a series of sub-actions with relevant parameters, such as a move action containing entity selection, destination setting, and the parameters such as the motor formation and motor speed. Unit action can be expresses as follows:

A = {t, p}

where t is the action type to identify the action, such as move, attack, scout, occupy, etc.; p is an action represented as a composition of a function identifier $a_{0}$ and a sequence of arguments that the function identifier requires: $a_{1}$ , $a_{2}$ , …, $a_{n}$ . For instance, consider selecting several units by drawing a rectangle; the action instruction will be “select_rect,” represented as $select_rect : {(x_{1}, y_{1}), (x_{2}, y_{2})}$ .

4.1.3. Rewards

In the DRL environment, reward signals are feedback from a tactical wargame that instruct combat units to learn appropriate actions to adapt to changes. In general, feedback is delayed, not immediate, because most actions, with the exception of attacks and occupations, do not obtain immediate results.

At each step, the AI agent is rewarded based on the scores provided by the game, which are calculated per-unit and per-action. In ArmorCombat, the player with the highest score wins. To obtain these scores, a player needs to accomplish different goals in a given scenario. The reward signal includes the difference in the life value of the agent unit between time and time $t + 1$ ; also, in the military game context, capturing critical control points is important because it leads to higher scores than simply destroying a tank. The reward update equation is as follows:

R_{ss'} = w_{1} \sum_{k = 1}^{n} Δ h_{k} + w_{2} \sum_{k = 1}^{m} Δ f_{k}

(2)

where $Δ f$ represents the score the AI agent receives when occupying the control point; $Δ h$ is the damage to an opponent’s combat unit; $w_{1}$ and $w_{2}$ are the weighting factors; and the reward is the target approximation of these sub objectives.

4.2. Actor–critic framework

The AC³² method derives from the policy-based method; the policy gradient is the classic policy-based method, which can be represented by the following steps.³³

Step 1. Let state, action, and reward be represented as a sequence $τ$ ; usually after sequence $τ$ , reward r can be obtained as follows:

τ = {s_{1}, a_{1}, r_{1}, s_{2}, a_{2}, r_{2}, \dots, s_{T}, a_{T}, r_{T}}

(3)

Step 2. Calculate the gradient that represents the expectation of reward relative to parameter θ:

\nabla_{θ} J (θ) = E_{θ} [\nabla_{θ} \log π_{θ} (τ) r (τ)]

(4)

where $π_{θ} (τ)$ represents the probability of selecting strategy $π_{θ}$ . The logarithm of the probability is multiplied by the return $r (τ)$ , and gradient is the result.

Step 3. The action model is to update $θ$ based on episode $τ$ :

θ \leftarrow θ + η \nabla_{θ} J (θ)

(5)

Through the gradient method, θ is updated and the optimal strategy is expected.

The traditional gradient descent method has its drawbacks, that is, it only updates the parameters at the end of the episode. We extend $r (τ)$ in Equation (4) under the condition that the strategy gradient remains unchanged, and then the equation is further expressed as follows:

\nabla_{θ} J (θ) = \nabla_{θ} \log π_{θ} (s_{t}, a_{t}) ψ (s_{t}, a_{t})

(6)

Regarding $π_{θ} (s_{t}, a_{t})$ as the actor and $ψ (s_{t}, a_{t})$ as the critic, the equation constitutes a generalized AC framework. Due to the complexity of the tactical wargame, and the large space of action and state, we use the deep neural network to represent the combat state and obtain its feature map, and use different neural networks to output the action value. The AC framework is shown in Figure 4.

Similar to the general reinforcement model, the agent receives the combat state within the time step, and after analysis and processing, the feature map is obtained as the current state $s_{T}$ . The agent takes action $a_{T}$ according to the situation, and then gets feedback $r_{T}$ from the wargame. The goal of reinforcement learning is to maximize effect feedback.

Under the AC framework, the A2C³⁴ method is optimized for the critic based on the AC framework. The advantage function is calculated according to the critic module, then the result is fed back to the actor policy network, and the policy network is iteratively updated with the new input parameters. The critic network is based on state value function $ν_{θ} (s)$ , which is the expectation of $G_{t}$ , starting from state $S_{t}$ :

ν_{θ} (s) = E [G_{t} | S_{t} = s]

(7)

A2C adds the baseline, which can assist to generate positive and negative feedback, making it easier to achieve optimal result. The dominant function is expressed as follows:

A = r_{t + 1} + γ v_{θ} (s_{t + 1}) - v_{θ} (s_{t})

(8)

Then, the actor updates in the following way:

θ \leftarrow θ + η (r_{t + 1} + γ v_{θ} (s_{t + 1}) - v_{θ} (s_{t})) \nabla_{θ} \log π_{θ} (s_{t}, a_{t})

(9)

In ArmorCombat, in order to get better results, we have improved the algorithm. A is not the result of one frame, but rather the combined weighted result of several consecutive frames. The improved A2C algorithm is shown in Table 1.

Table 1.

Advantage-actor critic algorithm.

01	initialize: init state s₀, actor: $π_{θ}$ , critic: V(s), γ,
02	for t=0,1,2… do
03	choose action $a_{t} ~ π_{θ} (a_{t} \| s_{t})$
04	get state s_t ₊₁ ,r from Env
05	update critic network using s_t, r, s_t₊₁
06	set $r [steps]$ , $V [steps]$ every n steps for calculate $A_{π}$
07	calculate td_error $A_{π}$ using advantage method $A_{π} = E [r_{t + 1} + γ * V (s_{t + 1}) - V (s_{t})]$
08	update the actor network using s_t, a_t, $A_{π}$ $θ \leftarrow θ + η (r_{t + 1} + γ v_{θ} (s_{t + 1}) - v_{θ} (s_{t})) \nabla_{θ} \log π_{θ} (s_{t}, a_{t})$
09	$s_{t} \leftarrow s_{t + 1}, t \leftarrow t + 1$
10	end for

In the tactical wargame, AI commanders use the AC model to make decisions. The agent consists of two components: the actor module and the critic module. The actor module is the agent’s actuator, which enters the external state (s), and then outputs the action (a). The critic module values the module and, with each step, it obtains an action value according to action (a) and effect feedback (r), and affects the actor module. In this process, actors are constantly iterating, the reasonable probability of each action is obtained for each state, and the critic is also iterating, constantly improving the reward of every action.

4.3. Deep network architecture

As mentioned above, the tactical game contains information about the battlefield environment, our combat units, and the enemy’s combat units. This information cannot be stored separately in a table, and different neural network designs must be created for these needs. It is necessary to establish how to express different kinds of combat actions in the game, how to output actions with the deep neural network, and how to design the return value of actions. All of this requires a great deal of innovative work.

In the DRL research environment, AI commanders use the A2C approach to learn a tactical strategy through interacting with the outside world. Therefore, from the battlefield situation to the action strategy, there is a large amount of information, which is represented by the deep neural network. The battlefield environment, game state, and unit state are necessary information for the deep neural network training. Thus, they all have their own data structure, and we must pre-process the feature layer containing categorical values. We use the one-hot encoding technique to convert the categorical values to an integer representation to generate a unified data structure with a size 128×128×16. For example, there are three categories of combat units: middle, red, and blue. Three separate binary feature planes are used to indicate the unit category. The red camp can be expressed as one-hot vector (0,1,0), the blue camp can be expressed as (0,0,1).

The CNN can learn abstract features of images. The hex map can be viewed as an image; the camp, position, and ammo of the combat unit as well as the environmental information, such as the height map, visibility map, and other information, can be regarded as different feature layers of the image, and the CNN can obtain the characteristics of the battlefield situation. In the AC framework, the value estimation of game states and the selection of tactical actions are achieved through deep neural networks, including several CNNs and fully connected networks. The net contains six layers, each of which has its own weights to complete its own functions: three layers are convolutional and the rest are fully connected layers. We use two convolutional layers and two fully connected layers to obtain the value of evaluating the current battlefield situation.

Now we describe the overall architecture of the deep neural network, as shown in Figure 5.

Then, we use two convolutional layers to form the feature map; the first convolutional layer filters input information with a kernel of size 5×5 and a stride of 1 pixel, while the second convolutional layer takes the output of the first layer as input, and filters it with a kernel of size 3×3 and a stride of 1 pixel. The output of each convolutional layer was subjected to Relu nonlinear processing, and the final feature map with a size of 128×128×32 was obtained.

We divide the tactical action into two parts: the action ID and the spatial parameter of an action. The action ID locates the specific action function, and the parameter part determines the specific details of the action. Here is an explanation of how these two parts work.

The action parameter part: the feature map is used as the input to construct the convolution network, which contains a layer to predict spatial parameters.

The action ID part: using the feature map as input, we conduct a fully connected layer that is fed to an output of size 256. The output result of the first layer is taken as common input of the action network and value estimation network. Then, the action network uses a fully connected layer that is fed to a 20-way softmax, and produces a distribution of over 20 action types; the value estimation network adopts a fully connected layer. The parameters of all of the neural networks are shown in Table 2.

Table 2.

Neural network parameter.

Layer	Kernel size	Depth	Strides	Padding	Input_size	Output_size
Conv1	[5,5]	16	1	SAME	[128,128,22]	[128,128,16]
Conv2	[3,3]	32	1	SAME	[128,128,16]	[128,128,32]
Conv3	[1,1]	1	1	——	[128,128,32]	[128,128,1]
FC1	——	——	——	——	128×128×32	256
FC2	——	——	——	——	256	20
FC3	——	——	——	——	256	1

4.4. Running process of the A2C framework in ArmorCombat

In the process of agent decision-making, the AI commander is the main body of the learning module; the actor model is used to interact with the environment in the policy gradient mode, the actor model is evaluated by the critic model in the method of the value function, and the action strategy is constantly iterated. The framework is performed in the following steps, and is shown in Figure 6.

Step 1. Set up the learning environment and initialize the relevant parameters.

In ArmorCombat, the AI commander gets the battlefield state, and the information of the combat unit. The battlefield state includes the target, type, location, strike point, attack state, etc. The action of the combat unit includes entity selection, move, attack, and so on.

Step 2. The convolution neural network is used to obtain the feature map of the battlefield situation.

We conduct a neural network with two convolutional layers, which input the battlefield information and output the abstract expression of the state.

Step 3. Build the actor network and the critic network.

The actor network is implemented by the deep neural network, and the input parameters are composed of the current battlefield state and the actions of the combat units. The combat units have a large space for action and are classified into action sets. The actor sampling network generates the action distribution probability, and the action selection function outputs tactical action in the action set. We initialize the critic network and get the value estimation according to the situation.

Step 4. Get rewards and the value of the current situation, and estimate the advantage functions.

An agent performs tactical action, interacts with the simulation environment, and gets feedback from the environment. The scoring rules are set for the simulation environment, and the agent performs different tactical actions to strike the enemy’s targets, and get the corresponding damage results, which are treated as environment feedback.

Step 5. Set N step actions as one episode, and the advantage function will get td_error obtained by the return of the game and the current battlefield value, and then feedback td_error to the policy neural network.

Step 6. Update the input parameter of the actor neural network, including battlefield situation information, tactical action information, td_error, etc.

Update the critic neural network input parameter.

Step 7. Repeat steps 2–6. AI agents constantly interact with the environment, update the parameters of the depth neural network, and continuously optimize the action strategies to achieve the set goals.

5. The experiment

ArmorCombat provides a learning environment that allows AI agents to fight against themselves as well as humans. In this section, our combat experiments are based on rule-based AI and DRL-based AI. After thousands of battles, DRL is likely to gain the ability to win the game. In the training process of the DRL agent, the rule-based AI always plays the role of the adversary.

5.1. Initial condition

We have designed a tactical scenario based on the jungle terrain that assumes the balance of force, operational intentions, and operational development of the two sides of the battle. The two fought on a 128*128 hexagon map that could represent different topographical features, such as jungles, roads, and rivers.

AI commanders are divided into two camps: the red camp and the blue camp, each with a number of different types of combat units, such as tanks, vehicles, and infantry. In addition, different unit types have different weapons, and a combat unit can be equipped with different kinds of weapons, with different shooting distances and damage effects. When the enemy is in range, the combat unit selects the optimal weapon to shoot. The operational instructions of both combat units are maneuver, shoot, get on, get off, occupy, etc. The strength of each force is mainly measured by mobility, attack ability, scout ability, and health value. The comparison of combat forces is shown in Table 3.

Table 3.

Forces comparison between the red camp and the blue camp.

Camp	Unit type	Num	Weapon	Health	Mobility	Attack ability	Scout ability	Total score
Red	M60A3 tank	6	Sight gun (medium), light weapon, missile	3	7	9	10	114
Red	LAV vehicle	6	Quick-firer, light weapon, missile	3	6	6	12
Red	Infantry	6 (team)	Rocket tube, light weapon	3	3	4	15
Blue	T80 tank	6	Sight gun (large), light weapon	3	7	10	10	132
Blue	BMPIQ vehicle	6	Sight gun (small), quick-firer, light weapon	3	6	8	12
Blue	Infantry	6 (team)	Rocket tube, light weapon	3	2	4	15

5.2. Experimental design

We designed a combat experiment between the AI agent-based AC method and the rule-based AI agent and evaluated the effectiveness of the DRL method in commander decision-making. AI agents run their own decision-making procedure. We set up the AI agent for the red camp to use the DRL method and the blue camp to use the rule-based method.

Like most games, tactical wargames have a main game loop. The main logic for handling AI is in the game loop. For each game step, the DRL agent and the rule-based agent perform their logic respectively. The design framework of the main step is shown in Figure 7.

In each game loop, the rule-based agent analyzes the rules of all actions and units in the knowledge base, which requires programmers to spend a great deal of time and energy, and then execute actions according to the rules. As can be seen from Figure 7, the DRL agent has many differences; firstly, the DRL agent also needs to get the combat state, but it must also be formalized into a feature map as the input of the neural network. Secondly, the action does not come from the knowledge base, but from neural networks. Thirdly, the parameters of the neural network are continuously modified through the reward feedback mechanism. With the development of the training network evolving, a variety of changes can be seen in action. In contrast, the rule-based agent can only execute fixed action rules.

In the human–machine battle mode, it usually takes five rounds to complete the entire game in 30 minutes. Under our experimental conditions, it is difficult to train the DRL agent in machine–machine battle mode, so we reduce the execution time proportionally. In addition, we found that the DRL agent is not good at completing the game, so we deployed combat units around the control point and let the AI agent play a round, so the AI agent can complete combat in 60 seconds, which greatly facilitated the DRL algorithm training.

5.3. Algorithm verification

ArmorCombat simulates the entire process form marching to attacking and occupying the control point. AI commanders make decisions throughout the process.

In the combat experiment, we can observe the tactical strategy difference between DRL agents and rule-based agents. The DRL agent continuously explores different tactics, and its action strategy is flexible, which is very helpful for the commander’s decision-making training. In contrast, the rule-based agent performs the built-in script rules, and its action strategy is fixed, so if it plays against human players, the game result can be easily decided when the rules are identified. However, the trained DRL agent still has weaknesses. On the one hand, it will learn to choose the action that will yield a higher score, such as shooting and battling for a control point. On the other hand, this may also become its weakness.

As mentioned in Section 4, the reward is mainly composed of the destroying score and the occupation score, which can reflect the effect of DRL training. Taking the combat results of every 50 rounds as a sampling point, the winning ratio curve of agents is as illustrated in Figure 8. It shows three different indicators: total score, occupation score, and destroying score.

Figure 8(a) shows battles results between rule-based agents. The force contrast of two sides plays an important role in combat. If one side is very strong, the other side will never win, and the experimental results will not be credible. We set up the two camps to use the same rule-based AI, and verified the validity of the initial conditions through combat experiments. In the jungle terrain scenario, the comprehensive indicator for evaluating the strength of the camp is the total score obtained. After 5000 rounds of battles, the red camp maintained an average winning ratio of 0.44, so the strength comparison of the two troops was reasonable. The results also show that each camp has its own strength advantage. The red camp based on the occupation score maintains an average winning ratio of 0.53, so the red camp is more likely to occupy control points than the blue camp, but the blue camp has more lethal weapons and can get a higher destruction score in the battles.

Figure 8(b) shows battles results between the DRL agent and the rule-based agents. The blue camp still uses the rule agent, but the red camp uses the DRL agent, adopting the basic AC method. As can be seen from the figure, from the beginning to the end of the battle, the winning ratio of the red camp maintained an average winning ratio of 0.49 and was not improved, which also reflects that the DRL agent has not found a better strategy, and cannot gain an advantage over the rule agent in the game.

Figure 8(c) shows battles results between the DRL agent and the rule-based agents. The blue camp still uses the rule agent, but the red camp uses the DRL agent, adopting the A2C method. The experiment was played in 7200 rounds and took a total of 4 days. The program was carried out at 320W steps and 16W episodes. After 7200 rounds of combat training, the DRL agent’s destroying score and occupation score were improved, the winning ratio of the rule-based agent was up to 0.9, and the average winning ratio was maintained at 0.8.

ArmorCombat combines TensorFlow algorithm tools, and the results displayed using TensorFlow are shown in Figure 9. As can be seen from the figure, the DRL agent found a better strategy after 1500 rounds, which greatly improved its performance.

We carry out further data analysis, and Figure 10 shows the winning ratio contrast of the rule-based agent and the DRL agent. As can be seen from the figure, the winning ratio of the DRL agent obviously improved after about 1000 rounds of training. Figure 10(a) shows that although the winning ratio of the DRL agent based on the destroying score increased, it is at the same level as the rule-based agents. Figure 10(b) shows that the winning ratio of the DRL agent based on the occupation score is much better than that of the rule-based agent after training. As can be seen from Figure 10(c), the average winning ratio of the rule-based agent based on the total score remained at 0.45, while the winning ratio of the DRL agent rose to 0.85 after about 1000 rounds of training.

After several rounds of battle training, the performance of the DRL agent is better than that of the rule-based agent.

6. Conclusions and future work

As the battlefield situation changes, the commander must also dynamically adjust decisions, which can be seen as a sequential decision-making problem. Unlike solutions based on rules and reasoning, this paper uses the DRL method to solve problems, improving the adaptability and flexibility of AI commander’s decisions.

We developed a tactical wargame as a testbed for our research and proposed a decision-making approach based on the AC method for the AI commander. In terms of the DRL algorithm, we combined the CNN and the fully connected neural network to carry out a fine value and policy network architecture, and used different deep neural networks to get the feature map of the battlefield situation, expressing value and policy functions. Then the running process of the AC framework is designed. Finally, the effectiveness of the DRL algorithm is verified by combat experiments. In the combat experiment, the DRL agent is able to explore diverse actions and strike strategies, which is obviously better than the rule-based agent, proving that the AC method is an effective method to solve the commander decision-making problem.

This paper provides a reference for the development of the AI commander in the field of tactical wargames. The DRL agent makes decision and executes a series of actions, some of which are necessary but do not produce a direct and immediate reward, so the reward feedback mechanism is the key factor. Currently, the DRL method has good performance in the case of using the jungle terrain scenario, but not in other scenarios. We need to further improve the versatility of the algorithm, which will be reported in our future work.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

ORCID iD

Junfeng Zhang

Author biographies

Junfeng Zhang received the BS degree in information engineering from PLA University of Science and Technology, Nanjing, in 2005. In 2016, he obtained the MS degree in foreign military thoughts from the PLA Academy of Military, Beijing. He is currently pursuing the PhD degree in combat simulation at the Army Academy of Armored Forces, Beijing, China. His research interests include DRL and multi-agent systems, while his main research area is combat simulation.

Qing Xue graduated from Harbin Institute of Technology, Harbin, in 1984. He is currently a professor at the Army Academy of Armored Forces, Beijing. His research interests include system modeling, equipment combat simulation, and data mining.

References

Caicedo

Lazebnik

. Active object localization with deep reinforcement learning. In: O’Conner

(ed.) IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015, pp.2488–2496. NW Washington, DC: ICCV.

Guo

Lee

, et al. Action-conditional video prediction using deep networks in Atari games. In: Cortes

(ed.) International conference on neural information processing systems, Montreal, Canada, 7–12 December 2015, pp.2863–2871. Cambridge, MA: MIT Press.

Silver

Huang

Maddison

, et al. Mastering the game of Go with deep neural networks and tree search. Nature 2016; 529: 484–489.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning. Comput Sci 2015; 8: A187.

Lillicrap

Sutskever

, et al. Continuous deep Q-learning with model-based acceleration. In: Langford

(ed.) International conference on machine learning, New York, USA, 19–24 June 2016, pp.2829–2838. New York, NY: PMLR.

Guo

Sprague

Replication of human operators’ situation assessment and decision making for simulated area reconnaissance in wargames. J Defens Model Simulat 2016; 13: 213–225.

Bojarski

Congdon

. REALM: a rule-based evolutionary computation agent that learns to play Mario. In: IEEE Computer Society (ed.) Computational intelligence & games, Copenhagen, Denmark, 18–21 August 2010, pp.83–90. NW Washington, DC: IEEE.

Pal

Shiu

SCK

. Foundations of soft case-based reasoning. Hoboken, NJ: John Wiley, 2004.

Amilde

. Modeling the behavior of a hierarchy of command agents with context-based reasoning. J Defens Model Simulat 2018; 15: 369–381.

10.

Pokahr

Braubach

Lamersdorf

, et al. Jadex: a BDI reasoning engine. Multiagent Syst Artif Soc 2005; 15: 149–174.

11.

Shen

O’Hare

GMP

Collier

. Decision-making of BDI agents, a fuzzy approach. In: Wei

(ed.) International conference on computer & information technology, Wuhan, China, 14–16 September 2004, pp.1022–1027. NW Washington, DC: IEEE.

12.

Fox

WP.

Applied game theory to improve strategic and tactical military decisions. J Defens Manag 2016; 6: 147.

13.

Lecun

Bengio

Hinton

Deep learning. Nature 2015; 521: 436–444.

14.

Krizhevsky

Sutskever

Hinton

. ImageNet classification with deep convolutional neural networks. In: Bartlett

(ed.) Advances in neural information processing systems, Nevada, USA, 3–6 December 2012, pp.1097–1105. Red Hook, NY: Curran Associates, Inc.

15.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition. Comput Sci 2014; 4: 1–14.

16.

Girshick

Donahue

Darrell

, et al. Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 2015; 38: 142–158.

17.

Liu

Anguelov

Erhan

, et al. SSD: single shot multibox detector. In: Leibe

Matas

Sebe

, et al. (eds) European conference on computer vision, Amsterdam, The Netherlands, 11–14 October 2016, pp.21–37. Cham, Switzerland: Springer.

18.

Wade

. Creating surrogate models for an air and missile defense simulation using design of experiments and neural networks. J Defens Model Simulat 2019; 9: 1–12.

19.

Amato

Shani

. High-level reinforcement learning in strategy games. In: van der Hoek

Kaminka

(eds) International conference on autonomous agents & multiagent systems, Toronto, Canada, 10–14 May 2010, pp.78–82. Bologna, Italy: AAMAS.

20.

Glavin

Madden

. Adaptive shooting for bots in first person shooter games using reinforcement learning. IEEE Trans Computat Intell AI Game 2014; 7: 180–192.

21.

Hosu

I-A

Rebedea

Playing Atari games with deep reinforcement learning and human checkpoint replay. Los Alamos, USA: arxiv.org, 2016.

22.

Kröse

BJA

. Learning from delayed rewards. Robot Auton Syst 1995; 15: 233–235.

23.

Peng

Pang

Yuan

, et al. Continual match based training in Pommerman: technical report. CoRR 2018; abs/1812.07297.

24.

Wang

Zhu

, et al. Belief-state Monte-Carlo Tree Search for Phantom Go. IEEE Trans Computat Intell 2017; 10: 139–154.

25.

Liskowski

Jaśkowski

Krawiec

. Learning to play Othello with deep neural networks. IEEE Trans Game 2018; 10: 354–364.

26.

Takada

Iizuka

Yamamoto

. Reinforcement learning to create evaluation and policy functions using Minimax Tree Search in Hex. IEEE Trans Game 2019; 12: 63–73.

27.

Vinyals

Ewalds

Bartunov

, et al. StarCraft II: a new challenge for reinforcement learning. Los Alamos, USA: arxiv.org, 2017.

28.

Arulkumaran

Cully

Togelius

. AlphaStar: an evolutionary computation perspective. In: Ibanez

(ed.) The genetic and evolutionary computation conference, Prague, Czech Republic, 13–17 July 2019, pp.314–315. New York, USA: GECCO.

29.

Tian

Gong

Shang

, et al. ELF: an extensive, lightweight and flexible research platform for real-time strategy games. Red Hook, NY: Curran Associates Inc., 2017.

30.

Wang

Zhang

, et al. Leader-follower consensus for multi-agent systems with three-layer network framework and dynamic interaction jointly connected topology. Neurocomputing 2016; 207: 231–239.

31.

Kandath

Senthilnath

Sundaram

. Mutli-agent consensus under communication failure using Actor-Critic Reinforcement Learning. In: Sundaram

(ed.) 2018 IEEE symposium series on computational intelligence (SSCI), Bangalore, India, 18–21 November 2018, pp. 1461–1465. NW Washington, DC: IEEE.

32.

Konda

Actor-critic algorithms. Siam J Contr Optim 2003; 42: 1143–1166.

33.

Sutton

McAllester

Singh

, et al. Policy gradient methods for reinforcement learning with function approximation. In: Cortes

(ed.) Proceedings of the 12th international conference on neural information processing systems, Denver, USA, 1 October 2000, pp.1057–1063. Cambridge, MA: MIT Press.

34.

Degris

Pilarski

Sutton

. Model-free reinforcement learning with continuous action in practice. In: Samad

(ed.) 2012 American control conference (ACC), Montreal, Canada, 27–29 June 2012, pp.2177–2182. Denver, USA: ACC.