Validation of the finite state machine model for network simulation and a proposed testbed

Abstract

Under the Cyber Army Modeling and Simulation (CyAMS) program a new model has been created to efficiently model various cyber events. The finite state machine model allows for the modeling of applications by creating behaviors that map to properties of real world applications. The finite state machine model is originally implemented by utilizing the ns-3 parallel discrete event simulator and validated using data from an emulation testbed experiment featuring malware applications. Following the completion of the ns-3 validation work, the CyAMS behavioral simulator was developed to allow for larger scale networks to be modeled, while also allowing both the network and applications to be defined using a behavioral network. This allows for the creation of an accurate mapping between both the simulation and the emulation applications. Validation tests are then carried out to determine both the validity of the model and the potential scalability. Finally, a testbed is proposed that will combine behavioral simulation, traditional parallel discrete event simulation, and emulation. The creation of this end-to-end environment for cyber simulation will allow past, present, and potentially future cyber events to be modeled in order to help understand and potentially mitigate future malicious attacks.

Keywords

Simulation network simulation network modeling

1. Introduction

Network simulators have become an important aspect of network communication research, and it is imperative that new models are created to efficiently model emerging threats. There are many benefits of utilizing simulation when modeling networks, including increased control over the environment, reproducibility of results, and scalability.¹ In an increasingly connected world, the ability to model large-scale cyber events has never been more important. Under the Cyber Army Modeling and Simulation (CyAMS) program, a collaborative effort between the U.S. Army Research Laboratory (ARL) and the U.S. Army Communications-Electronics Research, Development and Engineering Center, Intelligence and Information Warfare Directorate (CERDEC, I2WD), development of a new behavioral simulator allows modeling of various cyber effects, including the propagation of malicious software. By leveraging existing network simulation technology, such as ns-3, as well as the new CyAMS behavioral simulator, the authors developed a model to stochastically simulate large-scale cyber events. Both the ns-3 simulation approach and the CyAMS behavioral simulation model have undergone a series of validation efforts utilizing data from a network emulation. In addition, several tests were conducted to test the scalability of the CyAMS simulator. Finally, a proposed testbed is introduced that will combine emulation, simulation, and existing technologies to create an end-to-end testing environment for cyber simulations.

2. Motivation

Communication over both wired and wireless networks has become increasingly common in both the military and civilian worlds. As such, maintaining secure networks free of malicious software has become a concern of industry, academia, and governments around the world. The ability to simulate extremely large-scale networks and determine how potential malicious cyber events affect the network infrastructure will prove invaluable in predicting and potentially preventing or mitigating future cyber-attacks. Creating a way to seamlessly merge emulation, packet-level simulation, and behavioral modeling will begin to address this problem.

The goal of the CyAMS program is to provide a behavioral modeling and simulation (M&S) platform for integrated human, malware, physical event, and machine M&S, allowing users to test and interact with a representative global cyberspace. To accomplish this, CyAMS seeks to utilize both emulated and simulated assets to support modeling of both large global-scale and tactical scale cyber events. Simulation will allow for the recreation of a system through models that accurately recreate the behavior of real world applications. Likewise, emulation will allow for modeling of hardware-specific applications, specific operating system exploits, or other internal applications to a computer system. By embracing both emulation and simulation, CyAMS can account for both modeling of real systems and more abstract networks. Also, by focusing on behavioral modeling, CyAMS will allow human and real world physical effects to be modeled and simulated. In addition, leveraging simulation will allow for CyAMS to perform well on a variety of different platforms, from small-scale 1000 node networks on a desktop workstation to multi-million node networks simulated using a high performance computing (HPC) cluster.²

3. Finite state machine model

One of the goals of the CyAMS model is to provide a way to accurately model software application behavior in an adversarial environment. Within the simulation, applications are defined as finite state machines, allowing for the creation of a set of behaviors to effectively describe the fundamental properties of the target software. Finite state machines have been successfully used in the past for modeling both systems and applications.³ State machines will allow applications to be broken down into a series of states that represent how the application will interact within the network. This enables each component of the application to be defined to accurately portray its behavior within the simulation. In addition, new behaviors can easily be added or modified, and they provide a way to break apart applications into manageable states. Finally, this model allows for state changes to be efficiently processed using parallel computing architectures.

The simulation model design begins by taking a look at existing models to determine the best path forward. A goal of the CyAMS project is to be able to accurately model malware and malware propagation and, for this reason, epidemic modeling utilizing the susceptible, infected, recovered (SIR) and susceptible, infected, susceptible (SIS) models has become an area of interest. In the work, Accelerating information diffusion in social networks under the Susceptible-Infected-Susceptible epidemic model by Kandhway and Kuri,⁴ the authors look at modifying these epidemic models to fit the spread of information via social networks. This type of epidemic model lends itself well to the creation of state machines, and for the CyAMS model this was expanded to the modeling of all applications, as well as systems.

Figure 1 shows how a simple ping application can be broken down and represented within the CyAMS model. Figure 1 shows five individual finite state machines and their interactions with each other within the simulation space. The first finite state machine represents the ping client node, which features five individual states, some of which have multiple outcomes. Once the ping client has been initialized it will contact the Domain Name Server (DNS) requesting a name look up, which is represented by another behavioral state machine. After the ping client receives confirmation from the DNS server state machine, the client will begin communication with the ping server node. This figure illustrates a simple example of how the finite state model can be used to represent an application within the simulation.

Figure 1.

Finite state machine representation for the ping application. DNS: Domain Name Server.

4. Model validation

The finite state machine model was originally implemented by utilizing the ns-3 parallel discrete event simulator (PDES). Using ns-3 to implement the new model will allow CyAMS to leverage a packet-level PDES, as well as begin with an already established and well documented simulator. ns-3 features a comprehensive codebase; in addition, a number of previous validation works have been carried out on the platform.⁵ This made the simulator a good choice for the first implementation of the model.⁶ ns-3 also allows users to parallelize the simulation to run across multiple compute nodes, which will be necessary when working towards the goal of modeling large-scale, complex networks.

4.1 ns-3 validation

After development and implementation of the finite state machine model within ns-3, the next step was to validate the results against an emulation testbed experiment. Resources at the National Cyber Range (NCR) were utilized to generate results from a number of different test cases on networks of varying sizes. NCR’s emulation testbed allows for the finite state machine simulation model to be validated against real hardware applications. Working together with the NCR, a series of cyber scenarios were developed to represent the initial test for the finite state machine model within ns-3. A total of five test scenarios were designed to validate the network simulation model. The purpose of these validation tests will be to determine if the finite state machine model can not only replicate software applications, but also model the behavior of the network. These trials range in size from approximately ~100 clients to ~15,000 clients. The largest of these scenarios is Trial E, which features ~15,000 clients and utilizes the entirety of the resources available in the NCR testbed. In future scenarios the number of clients will be expanded to showcase the scalability of the model. In this initial scenario, there exist two malware applications, each of which is represented as a finite state machine within the simulation. Each of these trials were run by the NCR for 2 hours or until all of the vulnerable clients were infected by malware.

4.1.1 Validation trials

The five original validation trials created by the NCR are Trials A–E. Each of these trials was designed to represent a part of a larger network, with Trial E representing the entire network of ~15,000 host clients. Figure 2 shows a representation of the relationship between the smaller trials and how they are used to create larger trials. Trial A contains 10 enclaves each of which contain ~10 host clients. Each of these clients connects via a central enclave router. The second smallest trial is B, which contains 10 Trial A-sized networks totaling ~1000 clients that are connected together via a circular routing pattern. In a similar way, Trial C contains five Trial B-sized networks also connected via a circular routing pattern. Trial D connects together two Trial C-sized networks, and the final and largest, Trial E, connects together three Trial C-sized networks. These routing patterns are based on pre-existing routing within the NCR emulation testbed. This topology is used in further validation tests as more experiments are created based on the original scenarios. In each of these trials, ~60% of the host clients per scenario contain a NCR emulated antivirus, with an additional ~17% of enclaves per scenario containing an active firewall. Each of the two malware applications originates from the same host client, which is present in each trial.

Figure 2.

Original validation scenarios: the network is configured with smaller trials as a subset of larger scenarios.

4.1.2 Validation malware

In each of the trials, only the two malware applications are generating traffic. While this does not account for realistic network congestion or human effects on the network, for validation purposes, no additional traffic was added to limit the variability of the experimental results to only modeling the malware propagation rates. Once the simulation begins, each of the malware begins to attempt to infect other host clients. To accomplish this, each malware application utilizes a set of 16 threads tasked with selecting internet protocol (IP) addresses from a large range. The first eight threads select IP addresses from a large range of addresses outside of the host client’s enclave. The second set of eight threads will select IP addresses from a smaller range of addresses that exist within the infected client’s own enclave. Exact ranges for the threaded IP address selection are detailed in Table 1. After the malware has made a valid IP selection, it will attempt to set up a connection between the two clients. In order for this connection to be created, there must exist a bidirectional route between the infector and destination client. This selection process is based on similar malware, such as Code Red or Slammer, which also use a scanning strategy to search for vulnerable machines. In addition, the set of threads that select and promote local propagation is also rooted in similar real world malware.⁷ If the malware is able to successfully infect a client, this client will then become a child of the infector. Both of the malware applications present in the scenarios can only infect a total of three other clients, either directly or via a child. This will only extend for one generation, preventing the malware from dying out after only a few infections. These propagation limits are based on the Stuxnet malware, which utilizes a similar three infection limit. Upon reaching three infected clients, the malware will stop propagating and exit, following the original design laid out by the NCR.

Table 1.

Original National Cyber Range emulation internet protocol address selection.

Thread number	Address start range	Address end range
0	123.0.0.0	123.0.255.255
1	123.1.0.0	123.1.255.255
2	123.2.0.0	123.2.255.255
3	123.3.0.0	123.3.255.255
4	123.4.0.0	123.4.255.255
5	123.5.0.0	123.5.255.255
6	123.6.0.0	123.6.255.255
7	123.7.0.0	123.7.255.255
8–15 (Within enclave)	123.x.y.0	123.x.y.255

The first validation malware (M1) is able to infect any clients that are not protected by an enclave firewall, even applications that contain an active antivirus. In addition, clients will also have any active antivirus disabled. The second validation malware (M2) can only infect clients that either have no active antivirus, or have had antivirus disabled. M2 is able to bypass active enclave firewalls to infect clients that are protected by M1. In each scenario, there are a number of clients that exist both behind an active firewall and contain an active antivirus; these clients are effectively immune from both M1 and M2.

4.1.3 Validation malware finite state machines

The simulation will model each of these applications as finite state machines. Figure 3 shows how each malware is broken down into a number of states, each representing how the application interacts within the simulation space. Once either of the malware applications becomes active, they will then attempt to connect to an existing IP address by utilizing the IP selection threads. After the malware has selected a valid client IP, the malware will attempt to set up a connection, and then propagate itself to the new client. By breaking down the malware applications into a set of states, the behavior of each application can be modified to closely match the real world application. This also allows for applications to easily be modified for future use. For example, M1 could be modified to not only disable antivirus, but also to bring down an active enclave firewall.

Figure 3.

State changes for the malware application. IP: internet protocol.

4.1.4 ns-3 simulation validation results

After constructing the scenarios within ns-3 by utilizing the finite state machine model, each trial is run under the same conditions as its emulation counterpart. Data is recorded from each trial run and checked against NCR emulation trials. To do this, the number of malware infections is measured at critical thresholds. Times are recorded for when each malware reaches 10%, 50%, and 90% of host clients infected and then compared against the emulation for accuracy. Finally, after the data is checked for accuracy, the logistic function is then used to fit the data to a curve for population growth.

Trial A, being the smallest of the validation scenarios, contains 10 enclaves with a total of 97 host clients. This is also the only validation trial that contains no active enclave firewalls. Due to the fact that this is a relatively small number of clients, the malware propagation in this trial is slower than the larger trials, with only ~10% clients infected 1500 s into the simulation. This also causes Trial A to have somewhat inconsistent results across multiple runs due to the random nature of the experiment. Figure 4 showcases the ns-3 simulation malware propagation, as well as its emulation counterpart.

Figure 4.

Trial A validation scenario. NCR: National Cyber Range.

Trial B, which is 10 times the size of Trial A, contains a total of 970 host clients. This is also the first trial that contains active enclave firewalls. It is shown in Table 2 that M1 reaches 50% infected at ~2000 s into the simulation, with M2 reaching 50% at ~2500 s. ns-3 simulation M1 reaches each critical threshold within ~300 s of its emulation counterpart. Figure 5 displays the results of the ns-3 simulation plotted directly alongside the emulation data. In Trial B and successive trials, infection rates for both M1 and M2 are close, with M1 propagating faster near the start and then M2 accelerating as M1 begins to reach 10% infected. This difference in time that it takes for the emulation and simulation malware to reach critical thresholds can be attributed to the random nature of the IP address selection. This is especially prevalent in the smaller trials, as there are only a small number of valid IP addresses assigned to clients, while the selection threads still choose from the entire range of IPs. As the trials become larger in Trials, C–E, the differences between trials will become smaller because of this same effect. However, there are still times, for example, in Table 2 Trial E M1 90% infected, where the simulation malware reaches 90% infected ~360 s ahead of the emulation malware. The random selection of IP addresses can cause some of this unpredictability between trials.

Table 2.

Trial infection times (seconds).

Trial name malware #	10% infected time (Sim, Emu)	50% infected time (Sim, Emu)	90% infected time (Sim, Emu)
Trial B malware 1	1362, 1130	2066, 1770	2754, 2340
Trial B malware 2	1892, 1856	2558, 2604	3210, 3177
Trial C malware 1	520, 499	884, 862	1246, 1414
Trial C malware 2	830, 786	1196, 1206	1562, 1640
Trial D malware 1	526, 499	867, 897	1206, 1391
Trial D malware 2	835, 626	1187, 999	1537, 1499
Trial E malware 1	595, 659	925, 1126	1253, 1613
Trial E malware 2	593, 713	1021, 1176	1445, 1673

Figure 5.

Trial B validation scenario. NCR: National Cyber Range.

Trial C contains a total of 4907 client hosts, comprising five Trial B-sized networks. Figure 6 is the plot of the ns-3 simulation as well as the emulation data from the NCR. Table 2 outlines the time at which each of the malware reaches the critical thresholds. At approximately 480 s into the simulation, M1 reaches 10% infected, with M2 reaching 10% at ~700 s. In addition, as the number of valid IP addresses increases, the overall randomness of the simulation also decreases, allowing for much closer times between the simulation and emulation experiments. In Trial C, both M1 and M2 reach each threshold within at most ~170 s of each other.

Figure 6.

Trial C validation scenario. NCR: National Cyber Range.

The final two trials, D and E, are also the largest of the validation trials. Trial D contains 9841 client hosts and shows a similar propagation pattern to the previous trials, shown in Figure 7. The trend continues, showing that both malware reach 50% infected within ~300 s of each other. In addition, both the simulation and emulation malware reach 10% for M1 only ~20 s apart, with M2 at ~180 s apart. M2 begins to fall behind the emulation at around 50% infected but does reach 90% infected only ~10 s behind the emulation.

Figure 7.

Trial D validation scenario. NCR: National Cyber Range.

Trial E is the largest of the original emulation validation trials and is detailed in Figure 8, containing a total of 14,838 host clients. This trial features the full network created by the NCR for the purpose of validating the finite state machine model. M1 is the most restricted within this trial as a total of 272 enclaves are protected by an active enclave firewall. The propagation rates for this largest trial are very close between each malware. Both M1 and M2 each reach 10% infected at ~500 s into the simulation. The simulation and emulation malware reach each critical threshold within at most ~300 s.

Figure 8.

Trial E validation scenario. NCR: National Cyber Range.

4.1.5 Logistic function

After completing analysis of the ns-3 simulation trials, the next step is to fit the data to a curve using the logistic equation (1) for population growth. In this equation L represents the maximum possible population, k represents the slope of the curve, x represents the current time step on the x-axis, and finally x_o represents the midpoint value of the curve. In order to calculate the curve, the initial values need to be set for each of these variables. L is set to the maximum number of possible infected clients in the scenario. Next, x_o is set to the approximate midpoint of the curve based on the gathered data. As the final step, an approximate k for the slope of the curve is calculated. Then, by utilizing (2) the error is calculated. $f$ (x_m) is the measured population value at time x with $f$ (x_c) being the calculated population value at time x. By using these two values and calculating and minimizing the sum of squares, a close fit for the data is generated:

f (x) = \frac{L}{1 + e^{- k (x - x_{0})}}

(1)

E^{2} = {(f (x_{m}) - f (x_{c}))}^{2}

(2)

The results from this method generate a curve fit for each trial, depicted in Figures 4 –8. Data in these charts is also shown with the NCR gathered data sets. Trial A M1 has an average slope of ~.69543 and Trial A M2 is at ~0.04541. With each successive trial the slope of the curve continues to increase. Trial B M1 is ~0.19075, with M2 at ~0.18802. Trial C shows the greatest increase with ~0.44824 and ~0.41813 for M1 and M2, respectively. The calculated slope for each of the trials begins to stabilize as the number of host clients increases. In the final two trials, it is expected to see similar calculated slope values of ~0.37991 and ~0.37923 for Trial D M1 and M2, and in Trial E, ~0.3942 and ~0.3319. This gradual increase in slope demonstrates the effect of the increase in propagation in the larger simulation trials due to the decrease in the number of invalid IP addresses. In addition, by confirming that the simulation data does in fact map to the logistic curve, it is reaffirmed that this scenario closely resembles that of an epidemic model, one of the original inspirations for the finite state machine design.

4.2. CyAMS validation

After completion of the ns-3 finite state machine model validation tests, the next step is to then use both the emulation and ns-3 simulation results to further validate the CyAMS behavioral simulation model. In addition, several new tests were derived from the original emulation tests as well as three that test the potential scaling ability of the CyAMS behavioral simulator. One of the problems with the ns-3 simulation is the resource cost needed to scale to a large number of nodes.⁸ The CyAMS behavioral simulator hopes to help alleviate some of the resource costs by utilizing a more streamlined approach.

There has been previous work pertaining to the scalability of the ns-3 simulator, which is important to take into account when formulating an approach to solve the CyAMS scalability problem. In Pushing the envelope in distributed ns-3 simulations: one billion nodes, Nikolaev et al.⁸ present results from a large-scale ns-3 simulation up to 1 billion nodes. This work includes details on how the experiments were conducted, as well as hardware requirements for the large-scale tests. This work helped to frame the problem for conducting CyAMS scalability work and provides insight into the success of other simulation tools in reaching large scale as well as potential for improvements that could be made over the current ns-3 architecture.

4.2.1 Validation trials

This validation work begins by using the previous ns-3 finite state machine work as a base for the future validation. As such, the CyAMS simulator will first attempt to reproduce results from emulation scenarios, trials A–E.⁹ These original tests push the limit of the emulation testbed; however, ns-3 is still able to effectively scale much higher, and because of this, a new set of scenarios were developed. Inheriting ideas from the original tests, while allowing for a much larger number of nodes, these new tests were constructed with scalability in mind. An additional seven tests, Trials F–L, will seek to further test the limits of the CyAMS simulator and determine the validity of the model.

Each of these new trials features some number of enclaves, each of which contains ~100 host clients. Borrowing from the original scenarios, in each trial, ~60% of clients per scenario also contain an antivirus capability, as well as ~17% of enclave routers containing an active firewall. Each scenario will also utilize the same two malware applications from the previous tests. Every enclave router is connected in a circular routing pattern and put into clusters of 256, which are then connected via another upper level router, allowing for communication between enclaves. A third top-level router connects each of these upper level routers in pairs, each managing ~50,000 clients. Figure 9 shows an example of what this new network topology will look like. Although this routing pattern may not be reflective of a real world topology, for validation purposes this is sufficient. One major difference between the original scenarios is the way in which IP addresses are selected. In the original scenarios this selection process was limited to a small number of possible IPs; in order to scale to a larger number of host clients, this process had to be redesigned. To accomplish this, the number of selection threads is increased from 16 to 24.

Figure 9.

Network layout: trials consist of multiple routers connecting client hosts together.

The first eight selection threads select from IP addresses from within the host client’s upper level router. The next set of eight threads select from IP addresses in the host client’s own enclave, and the final set of eight threads select from completely random IP addresses. This new IP address selection process is highlighted in Table 3. The first four of the new tests will be run by utilizing the validated ns-3 model, and feature ~150,000–1,000,000 host clients. The final three trials are J, K, and L; these trials will be used to test the scalability potential of the CyAMS behavioral simulator. The scalability trials contain, ~10⁷, ~10⁸, and ~10⁹ clients, respectively.

Table 3.

New internet protocol address selection.

Thread number	Address start range	Address end range
0–7 (Addresses within upper level router)	x.y.z.0	x.y.(z + 255).255
8–15 (Addresses within client enclave)	x.y.z.0	x.y.z.255
16–23	1.0.0.0	255.255.255.255

4.2.2 CyAMS validation results

The CyAMS data will first be compared to the previous results from the ns-3 simulation and the NCR emulation results. Table 4 displays the times for the CyAMS simulator to reach the critical thresholds for the malware propagation in trials B–E. Trial A was excluded from these results due to the inconsistent nature of the trial; however, for each additional trial the CyAMS behavioral simulation is able to match within ~100 s of either the emulation or ns-3 simulation results. The CyAMS behavioral simulation was able to closely match both the emulation and ns-3 simulation results for the original scenarios.

Table 4.

Trial infection times (seconds).

Trial name malware #	10% infected(CyAMS, ns-3, Emu)	50% infected(CyAMS, ns-3, Emu)	90% infected(CyAMS, ns-3, Emu)
Trial B malware 1	1208, 1362, 1130	1785, 2066, 1770	2357, 2754, 2340
Trial B malware 2	1690, 1892, 1856	2428, 2558, 2604	3088, 3210, 3177
Trial C malware 1	481, 520, 499	767, 884, 862	1058, 1246, 1414
Trial C malware 2	952, 830, 786	1219, 1196, 1206	1494, 1562, 1640
Trial D malware 1	495, 526, 499	884, 867, 897	1266, 1206, 1391
Trial D malware 2	681, 835, 626	1103, 1187, 999	1522, 1537, 1499
Trial E malware 1	556, 595, 659	1023, 925, 1126	1525, 1253, 1613
Trial E malware 2	689, 593, 713	1147, 1021, 1176	1564, 1445, 1673

CyAMS: Cyber Army Modeling and Simulation.

Once it was determined that the new simulator can effectively reproduce results from the original trials, the new set of trials (F–I) are used to further test the model on a larger number of clients. The first of these new trials is F, for which the results are shown in Figure 10. In this trial at ~10% infected, the CyAMS is slightly behind its ns-3 counterpart, and this trend is continued throughout the simulation. This trend also continues into the second trial G displayed in Figure 11, with CyAMS reaching ~50% infected for M1 at ~33 s behind ns-3 and M2 reaching the same threshold at about ~38 s after ns-3. However, in each of these two trials CyAMS is able to reach each critical threshold within at most ~100 s of the ns-3 simulation. Similar to the previous ns-3 and emulation trials, these scenarios are also subject to the random nature of the IP address selection process used for propagation. This again allows for the CyAMS trial to sometimes lag behind its ns-3 counterpart when reaching critical thresholds; however, there are cases where the opposite is also true. Ultimately the goal with each of these scenarios is to determine if each simulator is able to reach each of these critical thresholds at similar times, and recreate the behavior of each malware application according to the original emulation specifications.

Figure 10.

Trial F validation scenario. CyAMS: Cyber Army Modeling and Simulation.

Figure 11.

Trial G validation scenario. CyAMS: Cyber Army Modeling and Simulation.

Figure 12 shows the results from the third validation test, Trial H. In this trial, each simulation M1 reaches 10%, 50%, and 90% within ~15 s of each other. In this trial, the CyAMS M1 is propagating ~8–10 s faster than the ns-3 M1. Nevertheless, CyAMS M2 is behind the ns-3 simulation counterpart, reaching 50% infected ~27 s after, suggesting that the difference in propagation rates may be due to the random nature of IP address selection. Finally, in this trial each simulation M1 reaches 90% infected at ~255 s.

Figure 12.

Trial H validation scenario. CyAMS: Cyber Army Modeling and Simulation.

The final ns-3/CyAMS validation test is Trial I. This is also the largest of these scenarios, featuring ~1 million host clients. The results from this scenario are highlighted in Figure 13. Similar to the previous trials, the CyAMS simulator is again closely matching ns-3, with M1 reaching 10% infected with only ~10 s difference between the two simulations. Nearing the end of the simulation, M1 reaches ~90% infected with ~17 s between the two simulations, and M2 with only a 5 s difference. Due to the increase in the number of available clients in this scenario, the difference in time between each simulator reaching a given critical threshold is smaller than in previous trials. As the scenarios grow larger, the chance for the IP selection process to select a valid IP increases, and with this decrease in negative responses the propagation times become much closer. The times that each simulation took to reach each of the critical thresholds is outlined in Table 5.

Figure 13.

Trial I validation scenario. CyAMS: Cyber Army Modeling and Simulation.

Table 5.

Trial infection times (seconds).

Trial name malware #	10% infected(CyAMS, ns-3)	50% infected(CyAMS, ns-3)	90% infected(CyAMS, ns-3)
Trial F malware 1	113, 90	167, 183	226, 216
Trial F malware 2	136, 116	227, 217	274, 265
Trial G malware 1	173, 149	228, 195	307, 248
Trial G malware 2	230, 209	302, 264	372, 314
Trial H malware 1	143, 168	199, 207	255, 264
Trial H malware 2	240, 207	285, 258	326, 311
Trial I malware 1	172, 162	205, 201	239, 256
Trial I malware 2	235, 205	268, 259	301, 306

CyAMS: Cyber Army Modeling and Simulation.

4.2.3 CyAMS scalability results

The final three trials are utilized to test the scalability of the CyAMS simulator. One of the limitations of both emulation and simulation are the resources required to scale to large global-sized networks. One of the goals of the CyAMS behavioral simulator is to attempt to model these large-scale networks in a more efficient way by utilizing finite state machines and defining the behavior of the network.

Each of the scalability trials was conducted using a standalone system featuring 24 CPU cores with a total of ~500 GB of RAM. The full resources of this machine will be used in scaling to the 1 billion host client scenario, and each trial will make use of the full 24 CPU cores. The first of the scalability trials is Trial J. Trial J features ~10 million host clients, and when the scenario is running at the peak number of infected clients, reaches ~30 GB of RAM usage. Figure 14 shows the results from this scalability test and shows similar results to the previous trials. Trial K is the second trial, detailed in Figure 15, which contains ~100 million host clients, and when running on the system utilizes ~80 GB of RAM. As these trials continue to grow in size, so does the number of events generated by the malware application within the simulation. As result, it is important that events are handled in a timely manner in order to ensure that the system does not become over-saturated. The final and largest of the trials is Trial L, which is the 1 billion client scenario and makes full use of the resources available in the system. The results from this trial are detailed in Figure 16 and depict the growth in the number of infected clients over time. Similar to ns-3, some of the smaller trials, such as A–D are as shown Figure 14. The Trial J scalability test is able to run faster than real time; however, as the number of clients and generated events increase, so too does the time required to conduct these tests. This may be an issue that can be solved in the future with faster computing power.

Figure 14.

Trial J scalability test. CyAMS: Cyber Army Modeling and Simulation.

Figure 15.

Trial K scalability test. CyAMS: Cyber Army Modeling and Simulation.

Figure 16.

Trial L scalability test. CyAMS: Cyber Army Modeling and Simulation.

5. Testbed (emulation, ns-3 simulation, Cyber Army Modeling and Simulation)

After the completion of the initial validation, the next step is to combine all of these technologies to create a complete end-to-end testbed environment for cyber M&S. This testbed will combine emulation, the CyAMS behavioral simulator, the ns-3 simulator, and GPU (Graphics Processing Unit) technology. Each of these can contribute varying levels of both granularity and fidelity, depending on the problem at hand. If the goal is to create a model for observing malicious code on a large-scale network that is not necessarily well defined, utilizing CyAMS or a statistical model running within the GPU may be useful. However, if the goal is to model a real system, emulation may prove more useful. The work, Emulation versus simulation: a case study of TCP-targeted denial of service attacks by Chertov et al.,¹⁰ does a comparison of emulation and simulation when modeling a denial of service (DoS) attack. They found that the simulation had difficulty modeling specific bottlenecks within the hardware due to feature abstraction. Shortfalls like this were worth noting when talking about the effectiveness of utilizing simulation over emulation, as well as when designing the CyAMS simulator to help address some of these potential issues. While working through this process it is important to note potential shortfalls of simulation in a cyber environment and recognize the importance of including emulation as part of the testbed to help combat these issues. However, where this testbed will really shine is when combining these technologies to model many types of cyber events simultaneously. Creating a way to model existing technology using emulation that will interact with malicious attacks coming from a behaviorally defined large-scale network is one of the goals of this testbed. By combining these technologies, the end goal will be the ability to create cyber scenarios with both varying granularity and fidelity to satisfy both current and future problem sets.

Emulation will be used to produce high-fidelity results within the CyAMS testbed. The emulation testbed will use virtualization to recreate systems in higher detail than may be possible using simulation. The previous example of recreating a DoS attack is one area in which emulation may prove to be more useful than just simulation alone. In this experiment presented by Chertov et al.,¹⁰ both simulation and emulation were used to compare differences in results from a modeled DoS attack. This research showed that simulation had difficulties modeling bottlenecks generated from “…CPUs, buses, devices, and device drivers….” This highlights the fact that the use of both emulation and simulation will be useful when modeling possible large-scale cyber events. While emulation can be useful for modeling finer details that matter for something like a DoS attack, simulation could be used for modeling malicious software propagation, as defined previously. By bridging all of these capabilities, the goal will be to allow any cyber event to be efficiently and effectively modeled and simulated.

The CyAMS behavioral simulation will have a unique role to preform within this testbed. There are many potential problems that cannot be easily modeled in a traditional emulation or simulation workspace, and the CyAMS simulator will attempt to bridge these gaps. To do this, CyAMS will allow for behavioral modeling of human and physical events, as well as systems to be simulated in a seamless manner. In addition, the CyAMS simulator platform will allow for varying degrees of complexity, due to the fact that everything will be represented as a behavior state machine, allowing for problems to be as simple or complex as needed to meet the mission goals. Coupling this with emulation will allow for future systems or ill-defined problems to be simulated alongside well-defined systems to help predict or mitigate future cyber events.

In addition to utilizing ns-3 to model networks at the packet level, this testbed will also make use of ns-3’s real time scheduler and emulation capabilities. The goal is to make ns-3 the glue that connects high-fidelity emulation with the CyAMS behavioral simulation. One of the potential problems when combining emulation and simulation together is that one must execute in real time and the other in virtual or simulated time.¹¹ In Integrated network experimentation using simulation and emulation, Guruprasad et al.¹¹ express the importance of maintaining a clock that is consistent between each respective platform. ns-3 will help to solve this by allowing for simulated packets to be passed to other real or emulated networks via a TapBridge and its real time scheduler. By employing this feature, the goal will be to model any real systems or hardware using emulation and then abstracting out larger networks to be modeled in either CyAMS or ns-3. One of the main differences between the CyAMS project and previous work by Guruprassad et al. is that this work sought to embed simulation as part of an emulator, whereas with CyAMS the goal is to create several independent parts that have the capability to work together dependent on the problem set. By using the ns-3 emulation capability, these environments will be connected together to create an end-to-end testbed for cyber M&S. The hope is that this will allow for the modeling of many current and future cyber-related problem sets. A scenario that may not be easily modeled using a traditional emulation testbed can be done using simulation, and any simulation shortfalls can be handled by the emulated or real hardware.

The final part of this testbed will be to incorporate existing technologies to both increase parallelizability and overall customization of the simulation workspace. Incorporation of the GPU into the testbed is one of these goals. The finite state machine model was chosen specifically due to the fact that there is potential to transition this model to the GPU in the future. A many core architecture like a GPU could allow for multiple states and state changes to be processed in parallel, leading to an increase in efficiency and speed. The process of utilizing the GPU to achieve increased parallelization as well as speedup is detailed by Park and Fishwick¹² in An analysis of queuing network simulation using GPU-based hardware acceleration. Park describes in this work that by utilizing the GPU they were able to achieve 10× speed up with only a small increase in error over traditional simulation approaches. With CyAMS, the goal is to incorporate GPU technology to achieve similar results on a potentially larger scale.

6. Conclusion

A number of validation tests have been carried out under the CyAMS program. Firstly, the ns-3 finite state machine implementation is compared to the emulation data recorded by the NCR. After completing this validation, a series of tests are conducted utilizing the CyAMS behavioral simulator, including scalability tests reaching up to 1 billion nodes. Based on the results of the validation scenarios, it is clear that both the finite state machine model and the behavioral state model are capable of producing results similar to that of an emulated network. By combining all of these technologies, the creation of an end-to-end testbed for cyber simulations should be possible. In the future, this project will continue with the creation of the CyAMS testbed, featuring automatic buildout for both emulation and simulation. Once completed, this testbed will also need to be validated against past large-scale cyber events or other cyber scenarios.

Footnotes

Funding

Funding for this project was provided by The Assistant Secretary of Defense for Research and Engineering (ASD(R&E)).

Author biographies

Scott Brown is a contractor supporting the U.S. ARL, Computational Sciences Division. His current research focus areas include simulation sciences and epidemic modeling, as well as large-scale simulation. He graduated from University of Maryland, College Park, with a BS in computer science.

Harold Brown is a member of the U.S. Army Research, Development and Engineering Command (RDECOM) research team. His research spans integration of many aspects of global cyberspace, electromagnetic propagation, and other networked systems leveraging HPC and visualization methods. He graduated from The Ohio State University with a PhD and the Florida Institute of Technology with a BS, both through the respective electrical engineering departments.

References

Baldo

Requena-Esteso

Nñez-Martnez

et al . Validation of the IEEE 802.11 MAC model in the ns3 simulator using the EXTREME testbed. In: 2010 Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques (SIMUTools ‘10), Torremolinos, Malaga, Spain, 15–19 March 2010, Vol. 64, p. 9. Brussels, Belgium: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering). Available at: https://doi.org/10.4108/ICST.SIMUTOOLS2010.8705

Fujimoto

Perumalla

Park

et al . Large-scale network simulation: How big? How fast? In: 11th IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer Telecommunications Systems, 2003 (MASCOTS’03), Orlando, FL, 12–15 October 2003, pp. 116–123. IEEE.

Andrews

Offutt

Alexander

Testing web applications by modeling with FSMs. Software Syst Model 2005; 4: 326–345.

Kandhway

Kuri

. Accelerating information diffusion in social networks under the Susceptible-Infected-Susceptible epidemic model. In: 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), New Delhi, 24–27 September 2014, pp. 1515–1519. Piscataway, NJ: IEEE.

ns-3 Testing and Validation release ns-3.10. 5 January 2011, pp.1-26.

Brown

Henz

Brown

et al . Validation of network simulation model with emulation using example malware. In: MILCOM 2015 - 2015 IEEE Military Communications Conference, Tampa, FL, 2015, pp. 1264–1269. Piscataway, NJ: IEEE.

Moore

Paxson

Savage

et al . Inside the Slammer worm. IEEE Secur Privacy 2003; 1: 33-39.

Nikolaev

Banks

Barnes

et al . Pushing the envelope in distributed ns-3 simulations: One billion nodes. In: Proceedings of the 2015 Workshop on NS-3 (WNS3’15), Barcelona, Spain, 13–15 May 2015, pp.67–74, New York: ACM.

Brown

Russell

et al . Validation of network simulation model and scalability tests using example malware. In: MILCOM 2016 - 2016 IEEE Military Communications Conference, Baltimore, MD, 1–3 November 2016, pp.491–496. Piscataway, NJ: IEEE.

10.

Chertov

Fahmy

Shroff

. Emulation versus simulation: A case study of TCP-targeted denial of service attacks. In: 2nd International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, 2006 (TRIDENTCOM’06), Barcelona, Spain, 1–3 March 2006, pp.10,325. IEEE. doi:10.1109/TRIDNT.2006.1649164.

11.

Guruprasad

Ricci

Lepreau

. Integrated network experimentation using simulation and emulation. In: First International Conference on Testbeds and Research Infrastructures for the DEvelopment of NeTworks and COMmunities, 2005, pp.204–212. Piscataway, NJ: IEEE.

12.

Park

Fishwick

. An analysis of queuing network simulation using GPU-based hardware acceleration. ACM Trans Model Comput Simul 2011; 21(3): 18. New York: ACM.