Machine learning-based discrete event dynamic surrogate model of communication systems for simulating the command,control,and communication system of systems

Abstract

Command and control (C2) and communication are at the heart of successful military operations in network-centric warfare. Interoperable simulation of a C2 system model and a communication (C) system model may be employed to interactively analyze their detailed behaviors. However, such simulation would be inefficient in simulation time for analysis of combat effectiveness of the C2 model against possible input combinations while considering the communication effect in combat operations. This study proposes a discrete event dynamic surrogate model (DEDSM) for the C model, which would be integrated with the C2 model and simulated. The proposed integrated simulation reduces execution time markedly in analysis of combat effectiveness without sacrificing the accuracy reflecting the communication effect. We hypothesize the DEDSM as a probabilistic priority queuing model whose semantics is expressed in a discrete event systems specification model with some characteristic functions unknown. The unknown functions are identified by machine learning with a data set generated by interoperable simulation of the C2 and C models. The case study with the command, control, and communication system of systems first validates the proposed approach through an equivalence test between the interoperable simulation and the proposed one. It then compares the simulation execution times and the number of events exchanged between the two simulations.

Keywords

Command control and communication mobile network discrete event dynamic system probabilistic discrete event systems specification nonlinear autoregressive exogenous model

1. Introduction

Command, control, and communication (C3) are the keys to managing the battlespace and to accomplishing combat operations and missions. In antiballistic missile warfare, for example, radars and effectors are often separate from a command post. Without proper communication, the effectors do not get targets assigned, and the decision nodes do not get the information they need to command and control the execution.^1,2 In this way, the elements of C3 affect each other in an interactive manner. Thus, these requisites should be simulated together to analyze combat effectiveness in network-centric warfare (NCW).

When explicitly simulating communications outside of war-game models—command and control (C2) system models as a broad term— interoperation is a well-known approach.³ This method enables the separate simulation of communication factors within the C3 simulation in the form of a federated simulation. In other words, the C3 federation contains multiple simulation components: the C2 federate and the communication (C) federate. These components are interconnected via an interoperation standard, such as high-level architecture (HLA) or a test and training enabling architecture.^4,5 Thus, the C federate, which controls the flow of information within the C2 federate, performs its tasks autonomously to allow for better decision-making in the overall C3 federation. Due to the representation of detailed behaviors and the guarantee of independence for each federate, simulation interoperation has been widely used in NCW simulation.^6–9

Although the approach via interoperable simulation of the C3 federation enables the independent description of detailed communication effects, it also requires extremely time-consuming simulation due to the computational complexity of the C federate and the overhead of interoperation middleware, such as the run-time infrastructure (RTI). To find a measure of combat effectiveness as a simulation goal, most NCW simulations have performed evaluations of possible parameter combinations as “what-if” analyses.¹⁰ However, such time-consuming effects make it difficult to conduct an analysis of combat effectiveness through various simulation parameters.

In this context, this study focuses on how to reduce the execution time of a simulation while ensuring the accuracy of its output (e.g., combat effectiveness) for NCW. The C2 federate, which has been widely regarded as a discrete event system, simulates most combat situations in the battlefield.¹¹ The C2 federate requests communication effects to the C federate by sharing resources, and the C federate continuously responds to the requests based on the network conditions. Because the goal of our simulation is to analyze the change in combat power as an effectiveness index while considering communication effects, we have kept the C2 federate unchanged, whereas we have abstracted the C federate using a meta-modeling method.

To transform the C federate into an abstracted surrogate model, we found its two main properties according to the mobility and availability of network nodes: (1) dynamic properties and (2) discrete event-based simulation. To reflect these points, we concentrated on a new approach via an integrated discrete event simulation with C2 in detail but C in the abstract.

In recent years, some researchers have similarly tried to reduce simulation costs by abstracting the C federate within their C3 federations. Notwithstanding the practical contributions of this work, primary limitations remain.^12–16 Firstly, these researchers did not fully represent the dynamic system property in their meta-modeling.¹⁷ In other words, a current input always led to the same output, as in a static system, even though different outputs could occur for the same current input due to the dynamic property of the C federate. Next, some of the researchers did not consider the discrete event simulation aspect of their meta-modeling, instead focusing only on the input/output (I/O) relation of the C federate. Consequently, their meta-models cannot be used as a model for discrete event simulation with the C2 federate.

To overcome this weakness, this paper proposes a discrete event dynamic surrogate model (DEDSM) that enables representation of the dynamic system property for discrete event simulation. To this end, we formularize the DEDSM using an atomic model of discrete event systems specification (DEVS) formalism¹⁸; we then identify how each of the four functions of the atomic DEVS model are characterized in the DEDSM. From a behavioral perspective, the proposed DEDSM can be designed as a priority queue with stochastically generated outputs. Machine learning techniques are also used to complete the behaviors. Specifically, end-to-end delay and packet-delivery ratio (PDR), which are the most significant factors in the communication network, are learned using the nonlinear autoregressive exogenous model (NARX).^19,20

Using the proposed meta-modeling method, we carried out an empirical analysis of the C3 federation, which consists of two federates: a C2 federate containing military operations of various combat entities, implemented through DEVSim++,²¹ and a C federate that includes the depiction of a mobile ad hoc network (MANET) with Network Simulator 3 (NS3).^22,23 We used the HLA/RTI for interoperation between the two federates. In this case study, we assumed that the simulations of the two federates were validated; thus, we concentrated on validation of the DEDSM of the C federate. For evaluation in terms of accuracy and performance, we first conducted an equivalence test between the C federate and its meta-model. Then, we compared the models in terms of execution time and the number of process events. Consequently, the experimental results show that both the execution time and the number of events are reduced by satisfying the equivalence test within acceptable ranges. Finally, we expect that our study will provide an alternative way for users to conduct simulation-based analyses of various scenarios, including the use of C2 parameters in interoperation environments that require long execution times.

This study is organized as follows. Section 2 describes the background of the problem; Section 3 introduces and compares previous works that are similar to our study. Section 4 explains how to apply the proposed DEDSM to the abstraction process of the C federate. Next, Section 5 discusses the experimental results from the perspective of accuracy and performance by comparing the C3 federation and an integrated simulation with the abstracted model. Finally, Section 6 presents our conclusions.

2. Background

Before moving to the central part of our work, we clarify some issues so that readers will understand the study’s overall context. This section provides background knowledge regarding the C federate of the C3 federation, especially its roles and properties.

2.1. Role of the C federate in the C3 federation

In a MANET, each node is free to move in any direction. Thus, links between nodes can change frequently. To accurately transmit data from source to destination, a node should continuously maintain the information that is required to route the traffic properly.

Figure 1 shows a simplified illustration of how the communication network, specifically the MANET, operates on the battlefield in a military application. Among six combat entities on the battlefield, a headquarters tries to send a launch command to an anti-defense missile. Each entity has a communication device that is mapped to a corresponding node; this device shares positional information between the battlefield and the communication network. In the network in Figure 1, solid lines represent currently reliable links, and the two types of dotted lines indicate the predicted links and directions. In addition, the shading of the nodes represents their available status; that is, darker nodes have busier conditions. Whenever a node needs to send data to another node on its network, the data will be routed to find a proper path according to the network conditions; this path is like a map. Only a subset of the intermediate nodes can be chosen to go from source to destination.

Figure 1.

Real-world view of the battlefield and its communication.

For example, given a normal situation in which all the nodes remain motionless, source node A would transmit data to destination node F with the routing path $\bar{ABCEF}$ through the immediate nodes B, C, and E. If we assume that node C moves upward and that nodes B and E move in directions $d_{BC}$ and $d_{EC}$ , respectively, the links $\bar{BC}$ and $\bar{CE}$ will still be reliable, and the path will not change. However, if B and E move in opposite directions against C (i.e., $d_{BD}$ and $d_{ED}$ ), $\bar{BC}$ and $\bar{CE}$ will no longer be available, and the routing path will be revised to contact approaching node D: $\bar{ABDEF}$ . Finally, if the two nodes move in different directions, the routing path will be chosen according to the availabilities of the immediate nodes. In the case of Figure 1, because D is busier than C, $\bar{ABCEF}$ might be a better choice than $\bar{ABDEF}$ .

The mobility and availability of each node give rise to transmission delays and failures in the communication network. When the state of a link changes from reliable to unreliable, one of the end nodes in this link will be newly selected as an active node. Thus, MANETs suffer from frequent link changes and failures, which in turn influence measures of effectiveness (e.g., combat power) in military applications.²⁴ For similar reasons, the realm of applications for communication effects is increasingly pervasive and includes military, aerospace, and manufacturing.^25–28

Next, we examine a system view of C2 and C federates based on a real-world view, as described in Figure 1. An overall battlefield situation is shown as a C2 federate, and the communication effect is represented by a C federate. We regard these two federates as a whole C3 federation. Fundamentally, these two federates exchange source and destination information to share positional changes. If a source entity wants to send a command to a destination entity in the C2 federate, the C2 federate will send a communication request message to the C federate via the interoperation environment. Then, the C federate will find an appropriate path and transfer the message to the destination node based on that path. After the transmission time for routing the message has elapsed, the C federate sends the communication response message to the destination entity in the C2 federate.

Because calculating the transmission delay and failure is the most significant role of the C federate, it usually generates two types of outputs regarding these calculations: (1) end-to-end delay and (2) the PDR.²⁹ The former indicates the average time taken by packets to arrive at the destination from the source node, and the latter is the ratio of the number of successfully delivered packets from the source to the destination node. Accordingly, the communication response message contains the effects of these two factors, which means the C federate should be classified as a discrete event dynamic system.

2.2. System taxonomy of the C federate: discrete event dynamic system

Systems are broadly classified into static and dynamic systems based on the presence of memory.³⁰ A system is said to be static if its output $y (t)$ depends only on input $u (t)$ at present time $t$ . Unlike a static system, the output $y (t)$ of a dynamic system will be affected by the inputs $u (τ)$ for $τ \leq t$ —in other words, not only the input at the present time $u (t)$ but also its past inputs $u (τ)$ will affect the system (for $τ < t$ ). A discrete event dynamic system is a (typically asynchronous) dynamic system in which state transitions are triggered by the occurrence of discrete events $e (τ)$ in the system (for $τ \leq t$ ). Typical examples of discrete event dynamic systems are flexible manufacturing systems, telecommunication networks, traffic control systems, and multiprocessor operating systems.³¹

In this context, we use simple examples to explain the C federate as a discrete event dynamic system. In Figure 2, we consider two simulation cases in which the C federate irregularly receives communication request messages from the C2 federate. Because messages from the C2 federate are scheduled dynamically as the simulation proceeds and because the C federate never expects them, the messages can be regarded as discrete events. Differences between the two simulations are the number of received events and the conditions that the nodes use to deal with the events. For example, the upper situation in Figure 2 indicates that the C federate receives an adequate number of events for handling. Thus, in state $S_{A}$ , it can successfully process output messages $y (t_{i})$ and $y (t_{j})$ for events $e (t_{i})$ and $e (t_{j})$ , respectively, with a slight transmission delay. In the lower situation of Figure 2, the C federate receives a tremendous number of events (i.e., communication request messages). This means that combat entities in the C2 federate frequently exchange messages between them under battle situations. In the case of $S_{B}$ , the response message $y (t_{i})$ is delayed more than twice as long as the response message of $S_{A}$ , and even $y (t_{j})$ is lost and so cannot reach the C2 federate.

Figure 2.

Two outcomes due to the discrete event dynamic property of the C federate: end-to-end delay and packet-delivery ratio.

The different outputs (i.e., end-to-end delay and PDR) against a current same input event are due to past events and not just the current event. In other words, a single input event, such as $e (t_{i})$ or $e (t_{j})$ in Figure 2, can produce different outputs at different times. Thus, the C federate can directly jump in time from the occurrence of one output to that of the next one because the outputs can spontaneously occur according to the internal condition. This is a typical characteristic of a discrete event dynamic system. In this respect, our main challenge in this paper is to maintain the property of the discrete event dynamic system when we abstract the C federate to enhance simulation costs.

3. Related work

Over the last decade, several studies have sought to abstract a communication network to speed-up C3 simulation. Even though these previous studies and our study all abstracted the communication system, the objective of abstraction (i.e., the target system of analysis) is different in our case. That is, the previous studies mainly focused on the analysis of the communication system, but we aimed to analyze the C2 system with the abstracted communication system. With these differences in mind, in Table 1, we summarize the three studies whose approaches are the most similar to our own.

Table 1.

Comparison of previous works for the abstraction of communication effects in defense simulation.

Previousstudy	Objective	Input, output, andregression approach	Static or dynamic system/discrete event modeling	Brief description
Porche et al.’s works^12,13	To quantify the marginal impact of networking as part of an effort to evaluate the concept of a network-centric operation	- Input: communication equipment specifications (such as data rate and frequency) - Output: PDR, end-to-end delay - Regression approach: response surface methodology	Static system/no	The network traffic (or load) is not considered in the meta-model. Not only is the system static; attending to the discrete event simulation is also difficult.
Miner et al.’s works^14,15	To provide the mechanisms and methodologies that enable the effects of communications system performance to be analyzed at the scale of a system of systems	- Input: communication network specifications (such as node density and traffic load) - Output: network availability - Regression approach: response surface methodology	Static system/no	Consideration of the network traffic is restricted in the meta-model due to the static form. Not only is the system static; attending to the discrete event simulation is also difficult.
Zang et al.’s work¹⁶	To analyze the characteristic parameters of military communication networks to meet the demands of the C3 system simulation	- Input: communication network specifications (such as the number of nodes and inter-arrival time) - Output: end-to-end delay - Regression approach: time-delay neural network	Dynamic system/no	The transmission loss (i.e., PDR) in the meta-model is not considered. Attending to the discrete event simulation is difficult, notwithstanding the dynamic system.

PDR: packet-delivery ratio; C3: command, control, and communication.

Firstly, Porche et al.’s works^12,13 were studied to quantify the marginal impact of communication networking as part of an effort to evaluate the concept of network-centric operation. In the previous works, communication network simulation was used to capture the dynamics of communication performance in closed-form expressions (meta-models).¹³ The meta-models were then embedded into force-on-force simulations for the warfighters’ overall effectiveness by evaluating how message loss and latency influence effectiveness in the selected scenario. Despite these studies’ practical contributions, in the abstraction process, their meta-models have a behavioral limit even without considering network traffic. They designed the meta-models with only I/O relationships (i.e., communication equipment specifications for inputs and message loss and delay for outputs).

Next, Miner et al.’s works^14,15 focused on a conceptual description to abstract a communication system from the perspective of system of systems-level architecture. They constructed a meta-model of the communication system using the response surface methodology (RSM). Unlike Porche et al., they reflected the network traffic as an input of the RSM.¹⁵ Nevertheless, they could not completely represent the traffic in the meta-model because the model has a static, not dynamic, form. As described earlier, network traffic is affected by the mobility and availability of the communication nodes; thus, it should be considered to reflect the consistent exchange of data between communication and war-game systems. Unfortunately, in these studies, communication performance was affected only by the current network traffic, and previous network traffic was not considered; furthermore, it is impossible to directly use the meta-model instead of the C federate because it is not the form of the discrete event model.

Finally, Zang et al.¹⁶ studied the characteristic parameters of military communication networks to meet the demand of the C3 system simulation. Simulation analysis in this paper focused on network performance itself rather than on overall combat effectiveness. The motivated meta-model of the military communication network was constituted, employing an uncertain time-delay neural network. Although the authors regarded the communication network as a dynamic system, the study suffers from an insufficient representation of communication behavior. For example, the authors did not consider transmission loss in their meta-model, only transmission delay; the form used in Zang et al.’s study also makes it difficult to directly attend to the discrete event simulation.

To summarize regarding the meta-modeling approach, although these authors tried to reduce the simulation time by abstracting the communication factors in the NCW simulation, no study has adequately covered the meta-modeling of a communication system while satisfying its discrete event dynamic property. That is, although research has been conducted to reflect the property of the dynamic system, no study has yet constructed a meta-model in the form of a discrete event model. Furthermore, regarding the transformation of the data into a meta-model, existing studies have constructed abstracted meta-models with acquired data by executing the standalone communication federate independently under certain scenarios; on the other hand, we use acquired data by executing the communication and C2 federate at the same time.

For this reason, the findings of the previous studies, when applied to our case, require the limitation of using meta-models in the I/O map of discrete event systems (i.e., the C federate) for two primary reasons. Firstly, the meta-model of the existing study and the discrete event model have different semantics. That is, the meta-model of this study is only a description of a function approximation based on given input and output data—no more than that. For this reason, this meta-model cannot be used as a model for discrete event simulation. Next, some previous research has shown no dynamics within the meta-model. As mentioned previously, this ultimately causes a sequence of different inputs to result in the same output. Thereby, in the C federate, the existing meta-model cannot handle inputs with different end-to-end delay times and PDRs in logical time order in the C federate. Therefore, we suggest a new type of meta-model, referred to as a communication discrete event dynamic surrogate model (C-DEDSM), to satisfy the above two constraints. Also, we show how the model can be transformed from the acquired data under C3 federation.

4. Discrete event dynamic surrogate model

In this section, we describe how the C federate is transformed into the C-DEDSM in the C3 federated simulation. We then propose a detailed modeling method for the C-DEDSM.

4.1. Overall procedure of federation abstraction

Prior to the description of the C-DEDSM, this section illustrates the overall procedure for the transformation of the C3 federation. As depicted in Figure 3, the process of federation transformation is similar to the general meta-modeling procedure, which consists of the following four major phases: (1) the design of the experiment for the C3 federation; (2) C3 federation execution and data acquisition for training; (3) DEDSM construction; and (4) standalone simulator construction.

Figure 3.

Overall procedure for transforming the command, control, and communication (C3) federated simulation to an integrated simulation. C2: command and control; C: communication.

In the first phase, we choose experimental points in terms of the C2 federate input (i.e., the C2 parameters). Because it is difficult to conduct the C3 federation with the entire design space due to execution time, we extracted some points from the entire design space. To extract the points without bias in the design space, we use various experimental designs, such as a face-centered central composite and a Latin hypercube.³²

The next phase executes the C3 federation using the extracted experimental points by storing (1) the data that the C2 and C federates exchanged during the simulation (i.e., communication requests and response messages) and (2) the output (i.e., combat power) of the C2 federate. The former contains information about the source and destination nodes (with sending and receiving times, respectively), which is used for training on the DEDSM in Phase 3. The latter is used to compare the output of the C3 federation and the abstracted integrated simulation for evaluation of the accuracy of the model.

After the simulation is completed, the next phase involves the construction of an abstracted model of the C federate based on the proposed DEDSM, using the data acquired from the federation execution (a detailed process of this phase will be described in the next section). In the process of the abstraction of the C federate, to deal with the aforementioned limitations, the proposed abstracted model should satisfy two conditions: it should be (1) a dynamic model with a dynamic system property and (2) a discrete event model for discrete event simulation. To be specific, the output of the model should be a function of its input history (or sequence); equivalently, it could be a function of a current input and a current state because a dynamic system such as a communication system can be represented in the form of the function. Secondly, the model should be a component of the discrete event model for an overall simulation model.

The last phase constructs a new standalone simulator by replacing the C federate in the C3 federation with the abstracted model constructed in the previous phase. In this phase, the C2 federate remains unchanged, but the C federate is changed into an abstracted model (i.e., a C-DEDSM). The C-DEDSM is integrated with the unchanged C2 federate so that the C3 federated simulation is transformed into an integrated simulation. After that, we compare the output, execution time, and number of processed events of the federation and integrated simulation from the perspective of accuracy and speed-up, respectively. In this respect, we define the situation as a problem and hypothesize a discrete event dynamic model in an atomic DEVS model; we then find four characteristic functions of the model.

4.2. Proposed communication discrete event dynamic surrogate model

As mentioned in Section 2, the C federate has two outputs: end-to-end delay and PDR. The dynamic system property influences these outputs. To represent the property of the end-to-end delay and PDR, this paper hypothesizes that the abstracted model of the C federate, which an atomic DEVS model describes, is a probabilistic priority queue in which the end-to-end delays associated with events are considered to be priorities. This paper also hypothesizes that the PDR is an influence in generating events because the C2 federate generates stochastic events in the mixed time.

Figure 4 shows a detailed description of the transformation procedures (Phases 2 and 3 in Figure 3). The first challenge (i.e., the dynamic system property) depends on how the end-to-end delay and PDR are calculated (i.e., predicted) when calling the calculateDelay and calculatePDR functions, respectively, as shown in Figures 4 and 5. Although this paper structurally decomposes the two outputs, the outputs can be provided in a single neural network depending on how we hypothesized the model.

Figure 4.

Detailed procedure of the transformation via the proposed discrete event dynamic surrogate model. C3: command, control, and communication; C2: command and control; C: communication; C-DEDSM: communication discrete event dynamic surrogate model; PDR: packet-delivery ratio; NARX: nonlinear autoregressive exogenous model.

Figure 5.

Description of the nonlinear autoregressive exogenous model (NARX) construction for the end-to-end delay and packet-delivery ratio (PDR).

This process consists of two subphases based on these questions. (1) What types of data are required for the prediction? (2) How should the prediction be conducted?

Although it is ideal for the prediction to use all the state information related to the target system (i.e., the C federate) that is to be abstracted, it is not easy to directly access that information due to the complexity of the target system and the inaccessibility of the black-box model.³³ Instead of being directly accessed, the state can be indirectly inferred through the input and output events related to it. In that respect, this study follows an indirect approach.

From the perspective of events, the states of the target system can be analogized by analyzing the relationships between the current output event and both the current and past input events. One of these methods is to use the time difference between the current and past input events (i.e., the inter-arrival time).³⁴ In fact, when and how often the entities in the C2 federate generate the communication request messages affect the C federate, so they can be indirectly analogized using the time difference. Based on this, in this paper, the inter-arrival time of events generated from the C2 federate to the C federate is the input for training. In case of the output, the end-to-end delay and PDR of each event are used. When acquiring data for each event, the PDR (unlike the end-to-end delay) is not a real number because each event is recorded as a transmission success or failure; therefore, we calculated the PDR as a ratio of 100 packets (i.e., one current and the previous 99 inputs events) at the sender side to the number of packets received at the destination.

As shown in Figure 4, after acquiring data on events from the C3 federation, two data sets are composed by pre-processing the data. To summarize, one data set (input: inter-arrival time, output: end-to-end delay) and another data set (input: inter-arrival time, output: PDR) against each event are used to train for prediction of end-to-end delay and PDR, respectively:

\begin{matrix} y [t] = f (y [t - 1], y [t - 2], \dots, y [t - N], \\ x [t], x [t - 1], \dots, x [t - N]) \\ (N : the number of tapped delay lines used in NARX) \end{matrix}

(1)

This paper applied the NARX with a closed-loop network architecture. While no feedback data is available in the model, we used an average value of raw data that was obtained from the interoperable simulation of C2 and C.³⁵

This method is used because it is suitable for representing the properties of a nonlinear dynamic system, which the communication system has, and has been commonly used for system identification in various methods, such as the autoregressive method, with or without exogenous input.³⁶ Therefore, to configure the atomic model in this way, the NARX is suitable for expressing the properties of the discrete event dynamic system.

Figure 5 contains a detailed explanation of how a NARX for calculating the end-to-end delay and PDR can be constructed with each data set. Firstly, the order of the equation (i.e., the number of I/O delays) is determined as the left-hand part; then, the form of the feed-forward neural network is determined from the equation as the right-hand part. Figure 6 shows the detailed structure of the feed-forward neural network of the NARX in Figure 5. It consists of input, hidden, and output layers. Each layer has neurons, which have a weighted connectivity with the neurons in the other layers. The values of the neurons in the hidden and output layers are acquired from the weighted sum of the neurons in the previous layer and from the activation functions ( $f$ and $g$ ), such as the sigmoid function, linear function, and so on, as in Figure 6. From the training, the weight is updated to minimize the errors of the desired output (i.e., end-to-end delay and PDR).

Figure 6.

Description of the feed-forward neural network.

Now, we are going to focus on the second challenge (i.e., discrete event simulation). Even though we have satisfied the dynamic system property, this is not a form for the discrete event simulation because it is only the I/O relationship. If the NARXs are used for the discrete time simulation, they can be applied to the simulation without any modification. However, because the C and C2 federates were involved in the discrete event simulation, the abstracted model should participate in the discrete event simulation. Therefore, to participate in the simulation with the C2 federate, the construction of a discrete event model is inevitable. Because the abstracted model is assumed to be an atomic DEVS model of a probabilistic priority queue, the process of constructing the abstracted model can be translated into the process of finding the four functions of the DEVS atomic model: δ_ext (external transition function); δ_int (internal transition function); λ (output function); and ta (time advance function). Of the four functions, we assume that two (i.e., δ_ext and λ) can be identified by analyzing the behavior of the general priority queue.

One of the remaining functions (i.e., ta) is associated with the remaining time for the output event; therefore, this function has a relationship with the end-to-end delay of the C federate. Therefore, identification of the function can be complemented by predicting the end-to-end delay. In addition, the other function (δ_int) describes the change of state for generating an output event. In this paper, the C federate does not always have output events for every input event, but the output events occur based on the PDR. Therefore, this study considers the internal transition function to be δ^p_int (the internal transition function with probability), with the PDR as the probability, by extending the DEVS model to include a probabilistic DEVS atomic model.³⁷ Therefore, identification of this function can be complemented by predicting the PDR. Because we have already identified the two required functions (i.e., calculateDelay and calculatePDR) in the first challenge, we can construct the atomic model without difficulty.

To summarize, the abstracted model can be represented as a probabilistic, priority-queue DEVS atomic model with prediction functions for the end-to-end delay and PDR. The two prediction functions are identified based on the data sets, which are acquired from the C3 federation and from applying machine learning techniques to the data sets. By affecting the δ^p_int and ta functions, the two characteristic functions are learned for the probabilistic DEVS atomic model.

Figure 7 shows a DEVS diagram of the abstracted atomic model based on the DEDSM, thus satisfying the requirements explained above. This model has the same inputs and outputs as the C federate because it is used as a surrogate model for that federate. To express a priority queue, this model has three states: (WAIT, q); (COMM, q); and (SEND, q). The first and second states are the states of the priority queue without and with messages received from the C2 federate, respectively; the third state is the situation in which the received message is sent to the C2 federate. In the (COMM, q) state, this model regards the minimum remaining time to send a message to the C2 federate (out of all of the messages in the queue) as the ta value.

Figure 7.

Probabilistic discrete event systems specification diagrams and specifications of the discrete event dynamic surrogate model.

To be specific about its operation logic, when receiving a message aiming to reflect the communication effects from the C2 federate in the (WAIT, q) state through δ_ext, this model calculates the end-to-end delay and PDR values using calculateDelay and calculatePDR. This atomic model stores the message with an output schedule time and the PDR by adding the current time and the end-to-end delay in a queue (q = {(remaining time for output, message ID, output schedule time reflecting delay time, PDR)}) with insertMessageInQueue. The queue is sorted by the remaining time of each message to be outputted, not the time of arrival (i.e., the received time for input). The remaining time would be decreased as simulation proceeds. Then, it refreshes the minimum value of the remaining time of the queue (i.e., minTA) using updateTA, and transfers to the (COMM, q) state. If it receives a message in the (COMM, q) state, it executes the same process. As each message arrives, the remaining times for all messages in the queue are updated and the list of messages in the queue is re-sorted. As such, the priority queue sorts the messages by a priority of remaining times for which the highest priority is given to the message with the smallest remaining time.

After determining the ta value in the (COMM, q) state, this model conducts δ^p_int to the (SEND, q) state based on the probability (i.e., the PDR for generating an event to the C2 federate). However, if the probability is not satisfied, this model changes to the (WAIT, q) state or remains in the current state according to the length of the remaining message in the queue, which is based on checkQueueLength. In the (SEND, q) state, this model immediately pops and sends the saved message to the C2 federate based on the popMessage and λ, respectively. After sending the message, if the remaining message exists after the length of the queue is checked with checkQueueLength, this model transfers to the (COMM, q) state and refreshes the ta value with updateTA; however, if there is no message in the queue, the model transfers to the (WAIT, q) state and changes the ta value to infinity. Based on the model structure, the abstracted model enables participation in the discrete event simulation.

To sum up, this paper proposes a new modeling approach, in that we combine the data model and the simulation model, which are already known concepts in the modeling and simulation (M&S) field, and create a model capable of discrete event simulation. Also, in the step of constructing the model, we propose a new development process to construct the model for discrete event simulation with the exchanged data executed between two federates, not from the standalone simulator.

5. Case study: command, control, and communication federation

The objective of this case study is to demonstrate whether integrated simulation with the proposed C-DEDSM can be substituted for the C3 federation from an accuracy perspective. The goal is also to determine how much time can be saved with the proposed method from a performance perspective. Therefore, we focus on two measurement indices (i.e., accuracy and simulation speed) by applying the proposed work to a C3 federation application. This section is organized as follows: the description of the C3 federation, the design of the experiment for the analysis of the above two indices, and the results of the experiment.

5.1. C3 federation description

Our case study deals with the C3 federated simulation for the army’s ground operations at an infantry-company level. This simulation’s ultimate objective is to analyze the combat power (i.e., enemy-survivability rate) against various C2 parameters (e.g., the time-related factors of decision-makers or the performance of time-sensitive weapons) in an environment with depictions of the communication effects of a real battlefield. To achieve this objective, this paper implements the C3 federation consisting of two standalone federators (i.e., the C2 and C federators), which have already been validated.³⁸

The C2 federate represents the military operational logics (i.e., doctrine, strategy, and organization) of forces and commanding officers accomplishing C2 operations. In the federate, information exchange between entities occurs through communication, and each piece of equipment used for communication that entities have conducts its function. To model the abovementioned content, the federate uses DEVS formalism that is highly compatible with model military systems organized hierarchically, and the designed models are implemented by DEVSimHLA, which supports the development of HLA-compliant simulators based on the sound semantics of the DEVS formalism.

On the other hand, the C federate focuses on the battlefield function of communication between the entities of the C2 federate. For this, this federate describes network equipment, which corresponds with the abovementioned combat entities of the C2 federate using the MANET based on a destination-sequenced distance vector (DSDV) routing protocol, which is usually used in military systems owing to its various merits.^24,39 To model communication equipment, this federate represents it as each network node. To realize the abovementioned content, the federate uses a NS3 discrete event network simulator that is publicly available for research and development, due to the merit of supporting various functions in an open-source environment.⁴⁰

To depict the combat entities of the C2 federate and the network equipment of the C federate in the same situation, we participate the two federates in a HLA-based interoperable simulation through a RTI. For a HLA-compliant simulation, we used a federation-object model (FOM). We formed one interaction including the communication message between nodes and the position information for each node. This C3 federation exchanged the interaction between the two federations, and it advanced the simulation time with the proper application program interfaces (APIs) in HLA/RTI.

In this C3 federation, we aimed to analyze the influence of various C2 parameters on the combat power considering communication effects, and Table 2 shows the parameters and description. We chose four parameters: the casualty radius of an indirect weapon, an intelligence fusion, a threat evaluation, and the weapon assignment time of C2. The former one and last three parameters are related to the performance of the time-sensitive weapons and time-related factors of C2, respectively.^41,42

Table 2.

Command and control (C2) federate parameters and their description of the command, control, and communication federation.

Parameter name	Parameter level	Description
Casualty radius (P_CR)	20, 21, …, 30 (m)	The casualty radius of indirect weapon
C2 intelligence fusion time (P_IF)	20, 24, …, 60 (s)	The time required for intelligence fusion in C2
C2 threat evaluation time (P_TE)	20, 24, …, 60 (s)	The time required for threat evaluation in C2
C2 weapon assignment time (P_WA)	20, 24, …, 60 (s)	The time required for weapon assignment in C2

5.2. Experimental design

Before building the proposed surrogate model, we conducted the C3 federation according to the procedure mentioned in Figure 3. In our four-dimensional C2 parameters in Table 2, the full factorial design size is 11 (for P_CR) × 11 (for P_IF) × 11 (for P_TE) × 11 (for P_WA) = 14,641, which requires a large number of experimental points and a long execution time, as mentioned in the motivation. In this situation, if we conduct 30 replicates against one experimental point for the statistical analysis, it requires approximately 650,060 hours because the average execution time per trial is approximately 1.48 hours. For this reason, we selected 34 experimental points from the entire design size using a face-centered central composite design (25 points) and a Latin hypercube design (nine points), as depicted in Table 4 and Phase 1 of Figure 3. Against the sample points, we executed the C3 federation 30 times per experiment point by acquiring the data exchanged between two federators for training and simulation results as in the left-hand part of Tables 4 and 5, as shown in Phase 2 of Figure 3. From the perspectives of accuracy and simulation-speed performance, we regarded the combat power (i.e., enemy-survivability rate and loss-exchange ratio), execution time, and number of events (processed in the C2, C federate, and C-DEDSM) as the effectiveness index. From the linear regression analysis as in Table 7 with data from Table 4, we were able to confirm that the four parameters have an impact on the combat power because the p-value of each parameter is less than 0.05, which demonstrates the need for analysis.

Also, Figure 8 shows the communication results according to the simulation time on the PDR and end-to-end delay from the C3 federation, as shown in Phase 2 of Figure 3. Based on the acquired data, we constructed a C-DEDSM for communication with optimized specifications as in Table 3, because too high or low dimension of the specification caused overfitting and time-consuming problems associated with training or caused difficulty with representing the dynamic system property. By substituting the C2 federate to the constructed surrogate model as in Phase 4, we transformed the C3 federation into an integrated simulation. To compare the simulation results from the C3 federation and integrated simulation, we assumed that we additionally chose 17 extra experimental points at random within the entire design space, which is the same with the design space of the experimental points for training, and we conducted the C3 federation and integrated simulation, respectively.⁴³ Table 8 illustrates the simulation results against the extra experimental points of two-paired data. Finally, the simulation environment for this case study is as follows. For the C2 federate, CPU: I5-3550 3.3 GHz, RAM: 4 GB, DEVSim++ v.3.1 were used. In the case of the C federate, we used NS3 v.3.18. These were interoperated by RTI 1.3-NG. The simulation progressed over 5000 seconds (i.e., when the combat power is sufficiently saturated). For the training, we used MATLAB neural network toolbox v.8.2.1.

Figure 8.

Communication effects in the command, control, and communication (C3) federation. PDR: packet-delivery ratio.

Table 3.

Description of the type and specification of the nonlinear autoregressive exogenous model.

Type	Specification
# of delays on input/output layer	3
# of neurons on hidden layer	10
Training algorithm	Levenberg–Marquardt
Hidden activation function	Hyperbolic tangent sigmoid
Output activation function	Linear

5.3. Experimental results

From the perspective of accuracy, because the ultimate objective was to use the proposed surrogate model to substitute the C3 federation into the integrated simulation in the analysis, we compared the respective combat power (i.e., enemy-survivability rate and loss-exchange ratio) against the same input, or experimental points.

Because the simple comparison (e.g., root-mean-square error) of the means of the output (μ_C3Fed, μ_IntegSim) from the C3 federation and integrated simulation does not provide statistical meaning, we first conducted a paired test consisting of the null hypothesis (H₀) and alternative hypothesis (H₁) against the combat power with experimental results from 51 experimental points (Tables 4 and 6). We assumed that the normality was guaranteed from the central limit theorem, which is required for the paired test:

H_{0} : | μ_{C 3 Fed} - μ_{IntegSim} | = 0, H_{1} : | μ_{C 3 Fed} - μ_{IntegSim} | \neq 0

Table 4.

Experimental results on accuracy of the command, control, and communication (C3) federation and integrated simulation for the training data set.

Exp.no.	Parameter set(P_CR, P_IF, P_TE, P_WA)	C3 federation		Integrated simulation		Difference btw. C3 federationand integrated simulation
		Enemy-survivability rate (%)	Loss-exchange ratio (%)	Enemy-survivability rate (%)	Loss-exchange ratio (%)	Enemy-survivability rate (%)	Loss-Exchange ratio (%)
1	(20, 20, 20, 20)	75.5658	73.6023	74.9177	75.4966	0.6481	−1.8943
2	(20, 20, 20, 60)	76.6564	70.2812	77.1708	68.6273	−0.5144	1.6539
3	(20, 20, 60, 20)	77.5514	67.6730	76.9650	69.2813	0.5864	−1.6083
4	(20, 20, 60, 60)	77.8498	66.5692	78.7243	63.9736	-0.8745	2.5956
5	(20, 60, 20, 20)	76.0288	72.2814	75.9259	72.4990	0.1029	−0.2176
6	(20, 60, 20, 60)	77.1502	68.7067	77.8601	66.5382	−0.7099	2.1685
7	(20, 60, 60, 20)	77.3354	68.1846	77.9527	66.4802	−0.6173	1.7044
8	(20, 60, 60, 60)	78.9918	63.1532	78.8477	63.6838	0.1441	−0.5306
9	(30, 20, 20, 20)	62.3971	113.7348	61.5638	115.9874	0.8333	−2.2526
10	(30, 20, 20, 60)	65.4938	103.9418	64.6091	106.6612	0.8847	−2.7194
11	(30, 20, 60, 20)	65.8230	102.9764	65.2263	104.8279	0.5967	−1.8515
12	(30, 20, 60, 60)	66.9033	99.9258	68.0658	96.1451	−1.1625	3.7807
13	(30, 60, 20, 20)	65.2881	104.4546	64.7325	106.5617	0.5556	−2.1071
14	(30, 60, 20, 60)	66.2449	101.8614	66.9342	99.9092	−0.6893	1.9522
15	(30, 60, 60, 20)	67.3148	98.6328	68.2305	95.7960	−0.9157	2.8368
16	(30, 60, 60, 60)	68.9403	93.6318	69.4753	92.0896	−0.535	1.5422
17	(20, 40, 40, 40)	76.8107	69.8344	77.4177	68.0238	−0.607	1.8106
18	(30, 40, 40, 40)	67.0885	99.2903	66.1523	102.3242	0.9362	−3.0339
19	(25, 20, 40, 40)	71.2654	86.6226	70.9465	87.4051	0.3189	−0.7825
20	(25, 60, 40, 40)	71.8724	84.9230	72.0988	83.9600	−0.2264	0.963
21	(25, 40, 20, 40)	70.4732	88.8744	70.4012	89.2278	0.072	−0.3534
22	(25, 40, 60, 40)	72.4486	83.0345	72.1708	83.9361	0.2778	−0.9016
23	(25, 40, 40, 20)	71.1831	86.7376	70.5864	88.6922	0.5967	−1.9546
24	(25, 40, 40, 60)	72.4588	83.1739	72.4074	83.0525	0.0514	0.1214
25	(25, 40, 40, 40)	71.5638	85.9869	71.4403	86.0293	0.1235	−0.0424
26	(27, 40, 32, 36)	69.7119	91.3758	68.9095	93.7246	0.8024	−2.3488
27	(24, 52, 24, 40)	72.8704	81.5340	72.5617	82.6514	0.3087	−1.1174
28	(29, 32, 44, 48)	67.8704	96.8076	66.9856	99.7028	0.8848	−2.8952
29	(23, 36, 48, 44)	74.5268	76.6732	74.3313	77.2222	0.1955	−0.549
30	(21, 56, 40, 32)	76.5741	70.6552	75.6584	73.3046	0.9157	−2.6494
31	(25, 44, 36, 56)	71.8313	84.9818	71.5432	85.7632	0.2881	−0.7814
32	(28, 24, 28, 28)	66.4095	101.3647	67.0165	99.6350	−0.607	1.7297
33	(22, 48, 56, 52)	76.3889	71.1229	75.3189	74.2890	1.07	−3.1661
34	(26, 28, 52, 24)	69.9794	90.6848	70.1646	90.0101	−0.1852	0.6747

The p-values were measured as 0.366 and 0.131 against the enemy-survivability rate and loss-exchange ratio, respectively. Although it cannot reject the null hypothesis, it cannot indicate the acceptance of the null hypothesis. Therefore, we conducted the equivalence test, which has been recently used in M&S.⁴³

To show the equivalence, we used the paired two one-sided test with the simulation result (Tables 4 and 6), and the below hypothesis shows its null hypothesis (H₀) and the alternative hypothesis (H₁). We assumed the normality required for the test based on the same reasons as before and conducted the equivalence test by changing the ε (i.e., the margin of error or equivalence threshold). Figure 9 illustrates the results:

H_{0} : | μ_{C 3 Fed} - μ_{IntegSim} | \geq ε, H_{1} : | μ_{C 3 Fed} - μ_{IntegSim} | < ε

Figure 9.

Experimental results of the equivalence test.

Figure 9 shows the variation of the p-value according to the ε (margin of error) for the enemy-survivability rate and loss-exchange ratio. Generally, we reject the null hypothesis and accept the alternative hypothesis if the p-value is less than the level of significance (α = 0.05).

In the case of the enemy-survivability rate, the 0.0239 p-value is measured at ε = 0.3, which means that the simulation result from the integrated simulation has a 0.3 difference at the maximum from the simulation results from the C3 federation with statistical significance. However, the equivalence is no longer guaranteed if the margin of error is smaller than 0.3 because the p-value is larger than the significance level. Nevertheless, ε = 0.3 is small enough not to interfere with the analysis of the relationship between the inputs and outputs when compared with the difference between the maximum and minimum values from the federation (=16.59): it is approximately 1.81%. From the perspective of the loss-exchange ratio, the p-value is 0.0497 at ε = 0.8 at a minimum and the difference between the maximum and minimum values from the federation is 50.58. The relative proportions of the margin of error and the difference is approximately 1.58%. Judging from the fact that the two relative proportions are small enough, we can regard the integrated simulation as being able to perform an identical role in the analysis of the combat power with the C3 federation within an acceptable range.

This small difference between the C3 federation and integrated simulation comes from the similar communication effects, as shown in Figure 10. The x-axis is the logical simulation time of the combat scenario, and the y-axis is the communication effects (i.e., the PDR and end-to-end delay). The figure shows that the C3 federation and integrated simulation have a similar tendency. Nevertheless, Figure 10(b) shows that the end-to-end delay in the C3 federation has higher variance than that in the integrated simulation due to the rare event of high delay in the C federate. This limitation is an inescapable problem in studies on model regression because the meta-model (i.e., data model) has a limitation in representing the property of the rare event.⁴⁴ Nevertheless, it does not have a significant enough an impact to hinder the analysis of the combat power.

Figure 10.

Comparison of communication effects of the command, control, and communication (C3) federation and integrated simulation: (a) packet-delivery ratio (PDR); (b) end-to-end delay.

From the perspective of simulation performance (i.e., speedup), we compared the total execution time from the C3 federation and integrated simulation against 51 experiment points (i.e., 1530 trials), as shown in Tables 5 and 8. In the case of the integrated simulation, we additionally considered the required time for data pre-processing and the machine learning training process of the simulation results from the 34 experiment points, in addition to the execution time for the integrated simulation. The execution time for the C3 federation was recorded as 2267.33 hours. On the other hand, in the case of the integrated simulation, 17.30 hours were measured bedsides 1.49 hours for data pre-processing and training. This ratio, approximately 131.05, indicates that the total execution time significantly decreased even though we considered the additional time required for the meta-modeling.

Table 5.

Experimental results on simulation speed from the command, control, and communication (C3) federation and integrated simulation for the training data set.

Exp.no.	Parameter set(P_CR, P_IF, P_TE, P_WA)	C3 federation			Integrated simulation
		Avg. # of events(C2 federate)	Avg. # of events(C federate)	Execution time(minutes)	Avg. # of events(discrete event dynamic surrogate model)	Execution time(minutes)
1	(20, 20, 20, 20)	2,761,211	463,061,907	2671	342,841	19
2	(20, 20, 20, 60)	2,793,042	468,526,889	2667	343,730	21
3	(20, 20, 60, 20)	2,788,379	469,808,555	2672	342,638	20
4	(20, 20, 60, 60)	2,814,625	465,921,668	2652	343,406	21
5	(20, 60, 20, 20)	2,771,414	464,660,766	2667	344,486	20
6	(20, 60, 20, 60)	2,796,831	462,345,520	2674	345,316	21
7	(20, 60, 60, 20)	2,798,263	455,777,218	2667	345,646	20
8	(20, 60, 60, 60)	2,820,965	466,064,454	2672	348,419	21
9	(30, 20, 20, 20)	2,593,128	469,353,518	2672	359,272	20
10	(30, 20, 20, 60)	2,645,842	469,489,962	2678	356,104	20
11	(30, 20, 60, 20)	2,652,852	471,084,020	2672	355,370	20
12	(30, 20, 60, 60)	2,686,517	471,901,632	2675	353,376	20
13	(30, 60, 20, 20)	2,629,244	467,265,837	2652	356,132	20
14	(30, 60, 20, 60)	2,649,533	466,600,198	2620	354,337	21
15	(30, 60, 60, 20)	2,658,745	471,168,203	2674	356,774	20
16	(30, 60, 60, 60)	2,699,518	470,543,357	2679	353,791	20
17	(20, 40, 40, 40)	2,805,864	464,340,878	2660	344,334	21
18	(30, 40, 40, 40)	2,657,111	467,931,858	2670	354,269	20
19	(25, 20, 40, 40)	2,722,447	468,325,721	2670	350,155	20
20	(25, 60, 40, 40)	2,739,485	464,843,793	2655	351,651	20
21	(25, 40, 20, 40)	2,719,824	468,574,680	2668	349,742	20
22	(25, 40, 60, 40)	2,749,443	470,173,181	2673	348,613	21
23	(25, 40, 40, 20)	2,703,722	467,271,895	2667	351,751	20
24	(25, 40, 40, 60)	2,738,722	466,382,828	2669	348,797	20
25	(25, 40, 40, 40)	2,729,648	466,405,607	2665	351,276	20
26	(27, 40, 32, 36)	2,680,614	469,373,894	2672	350,300	19
27	(24, 52, 24, 40)	2,726,214	465,729,311	2678	349,737	20
28	(29, 32, 44, 48)	2,684,842	467,970,231	2681	352,783	19
29	(23, 36, 48, 44)	2,760,696	465,546,059	2651	346,339	20
30	(21, 56, 40, 32)	2,780,338	464,606,816	2613	344,635	19
31	(25, 44, 36, 56)	2,723,050	468,603,017	2674	350,494	20
32	(28, 24, 28, 28)	2,644,576	467,224,164	2679	352,435	19
33	(22, 48, 56, 52)	2,799,544	467,428,133	2669	348,530	20
34	(26, 28, 52, 24)	2,696,868	469,037,206	2670	352,145	21

C2: command and control.

Table 6.

Experimental results on accuracy of the command, control, and communication (C3) federation and integrated simulation for testing the data set.

Exp.no.	Parameter set(P_CR, P_IF, P_TE, P_WA)	C3 federation		Integrated simulation		Difference btw. C3 federation and integrated simulation
		Enemy-survivability rate (%)	Loss-Exchange ratio (%)	Enemy-survivability rate (%)	Loss-exchange ratio (%)	Enemy-survivability rate (%)	Loss-exchange ratio (%)
35	(24, 40, 20, 40)	72.3148	83.5445	72.1811	83.6909	0.1337	−0.1464
36	(24, 44, 48, 48)	72.4691	82.7612	72.8807	81.7531	−0.4116	1.0081
37	(28, 24, 32, 28)	67.0370	99.4457	66.7078	100.2849	0.3292	−0.8392
38	(20, 28, 48, 28)	76.0288	72.1707	76.4506	71.0096	−0.4218	1.1611
39	(22, 52, 56, 40)	75.8230	72.7531	75.6687	73.2362	0.1543	−0.4831
40	(20, 60, 20, 20)	76.4609	71.0148	76.5123	70.8054	−0.0514	0.2094
41	(20, 32, 52, 44)	77.8498	66.5353	77.7572	66.8986	0.0926	−0.3633
42	(22, 48, 60, 24)	75.1235	75.1843	75.2778	74.4888	−0.1543	0.6955
43	(24, 56, 52, 44)	73.9815	78.3946	73.2099	80.8434	0.7716	−2.4488
44	(24, 56, 32, 48)	73.0761	81.0398	72.6543	82.1835	0.4218	−1.1437
45	(26, 20, 52, 24)	69.6811	91.4687	69.2181	92.7943	0.463	−1.3256
46	(24, 20, 24, 36)	71.8827	84.8051	72.1811	83.7765	−0.2984	1.0286
47	(28, 24, 36, 48)	67.4794	98.2367	67.7058	97.2786	−0.2264	0.9581
48	(22, 56, 56, 36)	75.9465	72.4925	75.7202	73.1185	0.2263	−0.626
49	(30, 24, 32, 40)	64.8971	105.8474	63.6111	109.6410	1.286	−3.7936
50	(20, 36, 32, 40)	77.7160	67.0910	76.7798	69.8383	0.9362	−2.7473
51	(28, 60, 24, 48)	68.9095	93.5572	68.8580	93.9760	0.0515	−0.4188

Table 7.

Command and control parameter influence analysis of the combat power in the command, control, and communication federation.

Enemy-survivability rate ~ 1 + P_CR+P_IF+P_TE+P_WA			Loss-exchange ratio ~ 1 + P_CR+P_IF+P_TE+P_WA
Parameter name	Estimated coefficient	p-value	Parameter name	Estimated coefficient	p-value
(Intercept)	94.591	1.115e-41	(Intercept)	16.015	8.5305e-08
P_CR	−1.0965	4.1709e-29	P_CR	3.3206	8.8384e-29
P_IF	0.02932	1.9212e-05	P_IF	−0.089608	2.4402e-05
P_TE	0.04863	2.3383e-09	P_TE	−0.14648	4.5025e-09
P_WA	0.034409	1.5586e-06	P_WA	−0.10486	2.1563e-06

Adj. R² = 0.987; p-value = 9.99e-28. Adj. R² = 0.986; p-value = 2.11e-27.

Table 8.

Experimental results on simulation speed from the command, control, and communication (C3) federation and integrated simulation for testing the data set.

Exp.no.	Parameter set(P_CR, P_IF, P_TE, P_WA)	C3 federation			Integrated simulation
		Avg. # of events(C2 federate)	Avg. # of events(C federate)	Execution time(minutes)	Avg. # of events(discrete event dynamic surrogate model)	Execution time(minutes)
35	(24, 40, 20, 40)	2,721,882	468,357,940	2673	345,950	20
36	(24, 44, 48, 48)	2,753,432	467,423,554	2668	349,091	21
37	(28, 24, 32, 28)	2,655,446	466,195,439	2670	353,299	20
38	(20, 28, 48, 28)	2,790,347	464,046,440	2674	344,060	21
39	(22, 52, 56, 40)	2,773,448	466,661,413	2677	348,022	21
40	(20, 60, 20, 20)	2,767,904	464,784,141	2670	342,872	20
41	(20, 32, 52, 44)	2,808,215	465,420,540	2675	344,475	22
42	(22, 48, 60, 24)	2,775,975	464,632,218	2669	349,839	22
43	(24, 56, 52, 44)	2,756,729	465,521,812	2645	346,525	21
44	(24, 56, 32, 48)	2,758,628	467,379,597	2653	348,893	22
45	(26, 20, 52, 24)	2,706,618	470,152,321	2675	349,231	21
46	(24, 20, 24, 36)	2,722,601	462,974,298	2683	351,014	21
47	(28, 24, 36, 48)	2,687,105	466,623,406	2666	353,110	21
48	(22, 56, 56, 36)	2,771,454	448,391,633	2674	343,547	21
49	(30, 24, 32, 40)	2,619,942	470,991,785	2681	355,477	20
50	(20, 36, 32, 40)	2,791,625	464,489,688	2660	345,364	20
51	(28, 60, 24, 48)	2,693,569	470,761,302	2673	354,463	21

C2: command and control.

For the detailed analysis of the simulation execution time, we measured the number of events processed in the C2 federate, the C federate, and the C-DEDSM, as shown in Tables 5 and 8. The ratio of events from the C2 and C federate (i.e., 170.99) shows that the simulation costs associated with the C federate are much higher than those associated with the C2 federate. Also, for the analysis of overheads due to interoperation middleware, we compared the ratios of the number of events processed in the C3 federation and the number processed in the integrated simulation. The former is the sum of the events from the C2 federate and the events from the C federate, whereas the latter is the sum of events from the C2 federate and the C-DEDSM. This ratio was approximately 152.46, which is larger than the above ratio of the simulation execution time. It implies that interoperation middleware is one primary factor causing the simulation time cost.

To summarize the experimental results from two perspectives, considerable simulation execution speed-up was achieved via transformation from the C3 federation to an integrated simulation. These results stemmed from (1) transformation from interoperation to an integrated simulation and (2) the simplified surrogate model via the proposed DEDSM approach. This paper measured these effects through the simulation execution time and through the number of processed events. Nevertheless, the accuracy decreased within the acceptable range enough to satisfy above the equivalence test, and it enabled us to substitute the integrated simulation for the C3 federation to conduct an analysis against various parameters in the C2 federate within the range of statistical significance.

Although this study applied to our developed C3 system, we expect this study to be applicable to other C3 systems, such as the interoperation of the OneSAF (i.e., war-game simulator) and the communication effect server (CES) (i.e., communication simulator), discussed by the Simulation Interoperability Standards Organization (SISO).⁶ Recently, many researchers, including the North Atlantic Treaty Organization (NATO), have paid attention to the command and control simulation interoperability (C2Sim) for the synthetic battlespace consisting of two systems: a C2 system, such as Système d’Information pour le Commandement des Forces (SICF), Integrated Command and Control (ICC), etc., and simulation, such as OneSAF, VR-Forces, etc.^45,46 Under this environment, our study is expected to help to describe the communication effects in the simulation. Also, even though this study was limited to the C3 system, we expect to apply other domains when abstracting a system between two systems considering the influence of another system other than the target system.

6. Conclusion

In NCW, because combat entities including C2 have exchanged information about the battlefield via communication, it is necessary to consider the detailed communication effects in analyzing combat power. For this reason, many researchers have used the communication simulator in an interoperation environment. However, this paradoxically causes a prolonged execution time.

Even though some studies have conducted an abstraction of the communication simulator in the defense M&S, they did not consider the dynamic system property of the communication simulator and represent it as a static form with a different semantic with the simulator (i.e., the discrete event model).

To overcome this weakness, this paper proposed a DEDSM of the communication simulator, which facilitates transformation of the interoperable simulation of the C3 federation to the integrated simulation. The proposed model, as a type of probabilistic priority queue, is represented by the DEVS formalism. Specifically, the time advance and internal transition functions cover the learning behaviors to train the end-to-end delay and the PDR.

In this paper, the current DEDSM only accounts for inter-arrival times of events in machine learning. To improve accuracy as well as reality we are working on machine learning for mobility of nodes, which will be embedded in the current DEDSM as a future work. We also will diversify the specifications of the NARX, such as the number of the I/O delay layers, and conduct an in-depth analysis of the relationship between the specifications and their implementation.

This paper compared the simulation speed and accuracy of the federation and integrated simulation, respectively. Our case study on C3 simulation shows that the execution time and the number of processed events are significantly reduced while maintaining enough accuracy to satisfy the equivalence test within an acceptable margin of error range. We finally expect this paper to be a step toward performing an analysis of various combat scenarios in an interoperation environment requiring a long execution time.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

ORCID iD

Kyung-Min Seo

Author biographies

Bong Gu Kang received his PhD in electrical engineering from the Korea Advanced Institute of Science and Technology (KAIST) in 2018. He is a research engineer at the KT institute of convergence technology and responsible for convergence projects related to information and communications technology. His research interest includes M&S of discrete event systems, digital twins, defense modeling and simulation, and interoperable simulation.

Kyung-Min Seo received his PhD in electrical engineering from KAIST in 2014. Currently, he is a research engineer at the Naval and Energy System R&D Institute of Daewoo Shipbuilding & Marine Engineering (DSME) Co., Ltd. He is responsible for naval system research and development projects related to submarines and unmanned underwater vehicles. His research interests include the discrete event system, cyber-physical system, system of systems, system interoperability, system integration, system monitoring and verification, defense modeling and simulation, and simulation analysis.

Tag Gon Kim received his PhD in computer engineering with a specialization in systems M&S from the University of Arizona in Tucson, Arizona, in 1988. He was an assistant professor of electrical and computer engineering at the University of Kansas, in Lawrence, Kansas, from 1989 to 1991. He joined the Electrical Engineering Department at KAIST in Daejeon, Korea, in the fall of 1991 and has been a full professor in the Electronic Engineering and Computer Science (EECS) Department there since the fall of 1998. He was the president of the Korea Society for Simulation (KSS) and was the editor-in-chief for Simulation: Transactions for Society for Computer Modeling and Simulation International (SCS). He is a co-author of the textbook Theory of modeling and simulation, Academic Press, 2000. He has published about 200 papers on M&S theory and practice in international journals and conference proceedings. He is very active in defense M&S in Korea. He is also a consultant for defense M&S technology at various Korean government organizations, including the Korea Ministry of Defense, Korea Defense Agency for Technology and Quality (DTAQ), the Korea Institute for Defense Analysis (KIDA), and the Korea Agency for Defense Development (ADD). He is a fellow of SCS and a senior member of the Institute of Electrical and Electronics Engineers (IEEE).

References

Tolk

Engineering principles of combat modeling and distributed simulation. Hoboken, NJ: Wiley, 2012.

Seo

Park

Lee

BJ.

Achieving data interoperability of communication interfaces for combat system engineering. IEEE Access 2017; 5: 17938–17951.

Seo

Hong

Kim

TG.

Enhancing model composability and reusability for entity-level combat simulation: a conceptual modeling approach. Simulation 2017; 93: 825–840.

IEEE Standard 1516:2010. IEEE standard for modeling and simulation: High level architecture – HLA framework and rules.

DoD. TENA-the test and training enabling architecture, https://www.tena-sda.org (accessed 24 October 2016).

Paz

Baer

. Communication effect server integration with OneSAF for mission level simulation. In: proceedings of the 2008 fall simulation interoperability workshop, Orlando, FL, 14–19 September 2008, pp.271–276.

Kang

Kim

TG.

Reconfigurable C3 simulation framework: interoperation between C2 and communication simulators. In: Pasupathy

Kim

Tolk

Hill

Kuhl

(Eds) proceedings of the 2013 winter simulation conference, Washington DC, USA, 8–11 December 2013, pp.2819–2830. Piscataway, NJ: IEEE.

Alberts

Garstka

Stein

FP.

Network centric warfare: developing and leveraging information superiority. DTIC Document, 2000.

Hofmann

MA.

Challenges of model interoperation in military simulations. Simulation 2004; 80: 659–667.

10.

Hong

Seo

Kim

TG.

Simulation-based optimization for design parameter exploration in hybrid system: a defense system example. Simulation 2013; 89: 362–380.

11.

Seo

Choi

Kim

et al . DEVS-based combat modeling for engagement-level defense simulation. Simulation 2014; 90: 759–781.

12.

Porche

Jamison

Herbert

. Framework for measuring the impact of C4ISR technologies and concepts on warfighter effectiveness using high resolution simulation. Santa Monica, CA: Rand, 2004.

13.

Porche

Isaac

Bradley

. The impact of network performance on warfighter effectiveness. Santa Monica, CA: Rand, 2006.

14.

Miner

Van Leeuwen

Smith

et al . Bridging the Information gap between diverse modeling and simulation tools. In: 14th annual systems engineering conference, San Diego, CA, 24-27 October 2011.

15.

Miner

Van Leeuwen

Welch

et al . Evaluating communications system performance effects at a system of systems level. In: IEEE military communications conference, Orlando, FL, 29 October–1 November 2012. Piscataway, NJ: IEEE.

16.

Zang

Jiang

Wang

et al . Research on motivated metamodel of military communication network. J Syst Simulat 2009; 21: 5562–5567.

17.

Kluever

CA.

Dynamic systems: modeling, simulation, and control. Hoboken, NJ: John Wiley & Sons, 2015.

18.

Zeigler

Praehofer

Kim

. Theory of modeling and simulation. 2nd ed. San Diego, CA: Academic Press, 2001.

19.

Kodratoff

Introduction to machine learning. San Mateo, CA: Morgan Kaufmann, 2014.

20.

Nelles

Nonlinear system identification. New York: Springer, 2002.

21.

Kim

Sung

Hong

et al . DEVSim++ toolset for defense modeling and simulation and interoperation. J Defens Model Simulat 2011; 8: 129–142.

22.

Henderson

Lacage

Riley

et al . Network simulations with the ns-3 simulator. In: SIGCOMM demonstration, Seattle, WA, 17–22 August 2008, pp. 527. New York: ACM.

23.

Kaur

Sharma

An overview of mobile ad hoc network: application, challenges and comparison of routing protocols. IOSR J Comput Eng 2013; 7: 7–11.

24.

Kuosmanen

. Choosing routing protocol for military ad hoc networks based on network structure and dynamics. Master Thesis, Helsinki University of Technology, 2002.

25.

Lee

Chi

SD.

DEVS/HLA-based modeling and simulation for intelligent transportation systems. Simulation 2003; 79: 423-439.

26.

Queck

Schunemann

Radusch

et al . Realistic simulation of v2x communication scenarios. In: Asia-Pacific services computing conference, Yilan, Taiwan, 9–12 December 2008, pp.1623–1627. Piscataway, NJ: IEEE.

27.

Mets

Ojea

Develder

Combining power and communication network simulation for cost-effective smart grid analysis. Commun Surv Tutorial 2014; 16: 1771–1796.

28.

Van Leeuwen

Oppel

Hart

High level architecture (HLA) federation with umbra and OPNET federates. Oak Ridge, TN, Sandia National Laboratories, 2004.

29.

Khan

Felemban

Qaisar

et al . Performance analysis on packet delivery ratio and end-to-end delay of different network topologies in wireless sensor networks. In: mobile ad-hoc and sensor networks, Dalian, China, 11–13 December 2013, pp.324–329. Piscataway, NJ: IEEE.

30.

Postalcioglu

Becerikli

Nonlinear system modeling using wavelet networks. In: Wang

Feng

(Eds) international symposium on neural networks, Chongqing, China, 30 May–1 June 2005, pp.411–417. Berlin: Springer.

31.

Branislav

MengChu

. Modeling and control of discrete-event dynamic systems. Berlin: Springer, 2007.

32.

Sanchez

Wan

. Better than a petaflop: the power of efficient experimental design. In: Rossetti

Hill

Johansson

Dunkin

Ingalls

(Eds) proceedings of the 2009 winter simulation conference, Austin, TX, 13–16 December 2009, pp.60–74. Piscataway, NJ: IEEE.

33.

Worden

Wong

Parlitz

et al . Identification of pre-sliding and sliding friction dynamics: grey box and black-box models. Mech Syst Signal Proc 2007; 21: 514-534.

34.

Amani

Kihl

Robertsson

. NARX-based multi-step ahead response time prediction for database servers. In: Abraham

Zomaya

Ventura

Yager

Snásel

Muda

Samuel

(Eds) intelligent systems design and applications, Córdoba, Spain, 22–24 November 2011, pp.813–818. Piscataway, NJ: IEEE.

35.

Basyoni

et al . Speed prediction from mobile sensors using cellular phone-based traffic data. IET Intell Transp Syst 2017; 11: 387–396.

36.

Wei

Zhao

Billings

et al , Fractional power NARX model identification using a harmony search algorithm. In: IEEE international conference on computational intelligence for measurement systems and applications (CIMSA), Tianjin, China, 2–4 July 2012, pp.102–107. Piscataway, NJ: IEEE.

37.

Seo

Zeigler

Kim

et al . Integrating web-based simulation on IT systems with finite probabilistic DEVS. In: D’Ambrogio

et al . (Eds) proceedings of the symposium on theory of modeling & simulation: DEVS integrative M&S symposium, Alexandria, VA, 12–15 April 2015, pp.173–180. San Diego, CA: SCS.

38.

Kang

Seo

Kim

TG.

Communication analysis of network-centric warfare via transformation of system of systems model into integrated system model into integrated system model using neural network. Complexity 2018; 2018: 1–16.

39.

Kang

Kim

. Abstraction on network model under interoperable simulation environment. In: Al-Begain

Bargiela

(Eds) 30th European conference on modeling and simulation, Regensburg, Germany, 31 May–3 June 2016, pp. 460–466. Berlin: Springer.

40.

Pan

Jain

. A survey of network simulation tools: current status and future developments. Report, Washington University, 2008, https://www.cse.wustl.edu/~jain/cse567-08/ftp/simtools/index.html (accessed 19 November 2018).

41.

Quigley

et al . An analysis of the open architecture warfare system domain model for surface time critical targets. Master Thesis Naval Postgraduate School, 2007.

42.

Moon

Carley

Kim

. Modeling and simulating command and control: for organizations under extreme situations. Berlin: Springer Science & Business Media, 2013.

43.

Bae

Kim

Moon

et al . Accelerated simulation of hierarchical military operations with tabulation technique. J Simulat 2016; 10: 36-49.

44.

Kim

Kang

Choi

et al . Data modeling versus simulation modeling in the big data era: case study of a greenhouse control system. Simulation 2017; 93: 579-594.

45.

Khimeche

C2-simulation interoperability benefits. NATO Document (STO-EN-MSG-141), 2015.

46.

Brook

C2Sim: C2 and simulation systems that have been implemented. NATO Document (STO-EN-MSG-141), 2015.