Modeling and simulation of future capabilities with an automated computer-aided wargame

Abstract

This article explores the development and application of an automated computer-aided wargame to establish high-level capability requirements and concepts of operations for future Navy unmanned aerial vehicles and unmanned underwater vehicles. The Joint Theater Level Simulation-Global Operations serves as the modeling environment, in which a computer-aided exercise models the impact of future intelligence, surveillance, and reconnaissance assets. Automating wargame simulations permits the replication of a large-scale exercise without the continued investment of support personnel and operating units. The environment enables experimentation that provides force planners with pertinent metrics to inform decision-making.

Keywords

Automated computer-aided wargame decision support tool joint theater-level simulation future capabilities

1. Introduction

Modeling and simulation (M&S) is widely used throughout defense and military communities as a training tool and is considered a key enabler for large multi-national coalitions, such as the North Atlantic Treaty Organization (NATO).¹ One such application is Computer-Aided Defense Planning (CADP), developed by Erdal Cayırcı and Lutfu Ozcakir in 2017.² The objective of using CADP is to streamline the defense planning process.² Part of this process is to use M&S for wargaming as an analytical tool to evaluate military capabilities or future requirements. M&S is used frequently to help in concept of operations (CONOP) development and experimentation. However, embedding it formally into the planning process still requires maturation.¹ M&S in wargaming requires a database buildup, planners, analysts, and time.

Successful force planning is contingent on appropriately answering the questions “How big?” and “How much?”.³ Considering this dichotomy between force structure and force costs, decision makers entrusted with building tomorrow's fleet must maintain a keen understanding of current and projected military deficiencies, while also paying due consideration to potential adversaries' evolving strategic objectives and military capabilities. Budgetary constraints further complicate this calculus, as a mismatch in strategic assumptions or force composition can be costly both financially and politically. Applying M&S as a means of experimentation to determine high-level functional requirements gives force planners a tool to gain insight into potential impacts of future technologies. Moreover, with an appropriate modeling environment, M&S can be utilized as a tool to develop and refine CONOPs for future technologies, allowing force planners to hypothesize and test viable ways in which to implement new fleet technologies. This process also bypasses the need to develop a new database for every wargaming scenario.

This article demonstrates the feasibility and a methodology of applying M&S vis-à-vis an automated computer-aided wargame (ACAW) to provide insight regarding high-level system requirements and CONOPs for future-fleet unmanned systems. This leads to the validation and verification of the requirements through designs of experiment and simulation. By examining the results of the simulation, this will demonstrate the use of the ACAW as a means to supplement future wargaming in CADP.

This article has four sections; it begins by providing motivation and introducing the foundation for the paper's content in Section 1. Section 2 describes the theoretical framework and methodology of the modeling scenarios and the design factors for the injected unmanned system assets within the computer-aided exercise (CAE), and segues to the experimental setup. Section 3 describes the data analysis and presents the experimental results. The paper concludes with a summary and insights in Section 4.

1.1. Joint Theater Level Simulation-Global Operations as a modeling environment

Rolands and Associates (R&A) developed Joint Theater Level Simulation-Global Operations (JTLS-GO) in 1983 and continue to refine the simulation’s air, ground, and naval operations based on user feedback.⁴ Being a theater-level simulation specifically designed to evaluate military strategies makes this program helpful in establishing initial CONOPs and requirements for future systems. Moreover, JTLS-GO has been adopted by sundry foreign and domestic defense organizations, including the Joint Warfighting Center, North Atlantic Treaty Organization (JWC NATO); Joint Staff Joint Warfighting Directorate (J7); and PACOM Warfighting Center (PWC), establishing credibility for the model's use as a test environment.

The simulation engine for JTLS-GO is the combat events program, or CEP. The CEP directs the actions and interactions for all air, ground, and naval units within the model. Traditionally, players interact with JTLS-GO via a web-hosted interface program, or WHIP, manually inputting mission sets and orders for units within the game. These orders are routed to the CEP, with feedback provided to the player by graphical updates in the common operating picture (COP) and formatted messages routed to the message browser component in JTLS-GO, as shown in Figure 1.

1.2. Framing scenario: Cobra Gold 2018

Cobra Gold is an annual military exercise held in Thailand that is sanctioned by the Pacific Warfighting Command (PWC). The latest iteration, conducted in February 2018, was attended by seven nations: the USA, Japan, South Korea, Thailand, Malaysia, Singapore, and Indonesia. Cobra Gold 2018 (CG18) consisted of a CAE that required a military-training audience to observe and accomplish key scenario events dictated by a master-scenario events list (MSEL). An opposition forces’ cell (OPFOR) serves as an active adversary with conflicting objectives, requiring the training audience to also react to simulated military threats.

The operational scenario for the model takes place in Pacifica, a fictional land mass southeast of Japan composed of six sovereign nations: Sonora, Mojave, Kuhistan, Arcadia, Isla Del Sol, and Tierra Del Oro; Figure 2 illustrates Pacifica.

The Sonoran invasion of land-locked Mojave causes regional destabilization, prompting response from a United Nations-sanctioned multi-national force (MNF). The goal of the MNF is to expel Sonoran invaders, maintain sea control in international waters off the Sonoran coast, and provide humanitarian assistance to the displaced Mojave refugees.

Using CG18 as the framing scenario to test future capabilities provides several advantages. Foremost, CG18 is a multi-day, operationally rich exercise that employs naval, army, and air forces. This provides ample opportunities to shape numerous vignettes to examine various CONOPs and systems within the model.⁵ In addition, defining key events in CG18 via the MSEL inherently provides a common model for shared insights. Moreover, the data output of CG18 is conducive to collection and analysis, providing a means to evaluate unit performance within the model.

1.3. Establishing alternative vignettes within CG18 to test future intelligence, surveillance, and reconnaissance capabilities

In the unmodified CG18 exercise, MNF reconnaissance aircraft, modeled as P-3s and P-8s, saw significant attrition by OPFOR air defenses while flying assigned patrol missions. Likewise, the MNF naval forces suffered casualties as a result of a degraded naval COP. Consequently, MNF intelligence, surveillance, and reconnaissance (ISR) capabilities were degraded, eroding the human players’ situational awareness of the operating environment. To address this operational shortfall, alternative vignettes of the CG18 scenario were created. These vignettes included additional ISR-capable unmanned aerial vehicles (UAVs) and unmanned underwater vehicles (UUVs) in the MNF force structure to patrol the Sonoran coastline and littorals, and provided a medium in which to gage the impact of future-fleet reconnaissance assets within the game.

2. Modeling and simulation of future capabilities

The establishment of vignettes leads to the modeling of a prototype that will fill the operational gaps. The stakeholders, then, must identify and choose the attributes (factors) for the proposed prototype. In JTLS-GO, this is done by entering values through the WHIP. After completion of the model, the prototypes can be injected into the simulation model. Varying the theoretically most important attributes implements a design of experiments (DOEs). Once the data is processed, the resulting information provides a decision maker with insights about desirable and undesirable factor combinations. In this study, the CONOPs for future UAVs and UUVs were examined to investigate their usefulness for reconnaissance missions.

2.1. UAV and UUV prototypes

The UAV prototypes are modeled after the MQ-4C Triton, which already exists in the JTLS-GO database. The JTLS-GO database provides pertinent information, such as the range, runway requirements, and operating altitude. Comparing the parameters associated with the UAV modeled in JTLS-GO with unclassified data from the Unmanned Systems Roadmap and other open-source information reveals a reasonably accurate representation of expectations for a high-altitude, theater-sized UAV. This process verifies the reliability of the JTLS-GO database.

The UUV is a relatively new concept and does not currently have units available in JTLS-GO. The simulation programmers plan to inject organic UUVs in the next version. As a result, an advanced UUV prototype was modeled after a mini-submarine. The UUV prototype has advanced sonar installed to pass the locations of detected units.

Creating and injecting UAV and UUV prototypes both occur in the Control-WHIP. Options available to the user include the name, unit prototype, naval qualification, squadron size, and home base. JTLS-GO acknowledges creation and placement via the message browser. The units are subsequently controlled in the individual player WHIPs.

2.2. Automating JTLS-GO

The ACAW uses two Python-coded (public-domain) software wrappers tailored to work with JTLS-GO: JTLS Farmer and JTLS Runner. Automating the simulation circumvents the graphical user interface (GUI) and precludes needing to have a game player present to manipulate or otherwise observe the scenario. More importantly, the automation of the exercise enables multiple simulations allowing for the application of statistical analysis. Figure 3 illustrates the process of simulating the unmanned systems to produce outcomes.

The first component, JTLS Farmer, starts the simulation, injects pertinent UAV and UUV mission directives, and runs the simulation using a sub-function called JTLS Runner. The user defines game start- and stop-times, as well as the number of game iterations. Controlling start and stop times allows several vignettes to be created from a single CAW, while conducting multiple iterations of a given vignette reduces estimate errors in the collected data.

The second component is JTLS Miner, which parses and collects macro data from the modified scenario by searching all the messages generated during the game and extracts those deemed important for discriminating UAV and UUV measures of effectiveness (MOEs). This data is then compiled into a standard comma-separated value (csv) file for analysis.

2.3. Design factors

In context of the model, design factors are parameters relating to the operating characteristics of the UAVs and UUVs. Altering these design factors provides a way to affect the response of the modeled systems.⁶

For the modeled UAVs, the design factors include mission altitude, time between sorties, and sortie size; a unique combination of these design factors comprises a design point. For example, three UAVs flying at 10,000 feet with 30 minutes between sortie launches would comprise one design point in an experiment. Similarly, the UUV design factors entail speed, quantity employed, and type of sonar (active or passive) used to find units of interest. Table 1 lists the quantitative design factors used for experimentation in the model, including the range of values that these factors may assume. The study team believes that these factors influence the behavior of the measures of interest described in the next section.

Table 1.

Experimental design factors.

Entity	DOE factor	Min	Mid	Max	Units
UAV	Altitude	10	30	60	kft
UAV	Number employed	1	2	3	UAVs
UAV	Time between sorties	0	30	60	Min
UUV	Speed	5	8	12	Knots
UUV	Number employed	8	12	16	UUVs
UUV	Sonar	Passive		Active

DOE: design of experiment; UAV: unmanned aerial vehicle; UUV: unmanned underwater vehicle.

2.4. UAV and UUV measures of effectiveness

A MOE is quantifiable data that evaluates the mission accomplishment of a system in its expected environment.⁷ John M Green⁸ further refines this definition by contextualizing MOEs as “quantifiable benchmarks against which the system concept and implementation can be compared.”

Intelligence messages and periodic reports generated by JTLS-GO serve as the basis for the quantitative data that reinforces the MOEs. Traditionally, these messages are read by the player via the message browser and communicate the number of high-value units (HVUs), such as anti-air weapons, naval vessels, or aircraft, discovered by the UAVs and UUVs. However, in automating the CAW and bypassing the GUI, the messages are instead collated and parsed by the JTLS Miner executable program for subsequent data analysis.

2.5. Design of experiment

For the purposes of this study, the DOE provides two primary benefits. Firstly, a DOE can allow the isolation of interactions within the model. Secondly, the DOE helps refine requisite UAV and UUV capabilities by identifying operational and system factors and combinations of factors that have the greatest impact on MOEs.

The team conducted two separate experiments, one for each system of interest, UUV and UAV. With regard to the factors and factor levels from Table 1, the team used a full-factorial design for each experiment. A full-factorial design explores the identified factor values to provide insight into the behavior of the modeled systems of interest, as they relate to those factors. In the case of the UAV, there are three design factors (altitude, number employed, and time between sorties), each with three identified levels to be examined, resulting in 27 unique combinations of the factor values. Similarly, there are three design factors for UUVs (speed, number employed, and sonar type), each with a different number of values to investigate. The result for the UUV experiment is 18 design points.

Because the team designated specific values to explore for each factor, these experiments were relatively simple, although computing time was a consideration because of the complexities of JTLS. However, had the team not discretized the values for say, Speed, there could have been an infinite number of factor value combinations that could have been investigated. Another continuous factor, Time Between Sorties, would have made experimentation much more complicated and required a much larger number of runs to examine the impact of the factors on the different MOEs. Works from Cioppa and Lucas,⁹ Sanchez and Sanchez,¹⁰ and MacCalman et al.¹¹ are just some designs that can handle a large number of factors and factor levels, as well as mixtures of discrete and continuous factors. For this study, it was unnecessary to implement these more sophisticated experimental designs.

2.6. Data analysis

Data analysis is accomplished using JMP statistical software. Specifically, the team used regression models, visualizations and experiment-driven optimizations to help quantify and qualify the results of the modified CG18 scenario.

3. Simulation results and analysis

The results from the altered simulation, with the added unmanned systems, came from 30 replications of each design point. The filtered data from the JTLS Miner program, through the use of an analysis program such as JMP, led to the visualization of the data allowing for analytical examination. The visualizations for this article include prediction profilers and partition trees; however, other statistical analysis products could also be used to explore different trends. The prediction profiler describes if and how the response variables correlate to the selected design factors. Partition trees are a data-mining tool that complements the information provided by a regression model and provides a rudimentary decision tree.¹² A cutting value within JMP determines the data splits that yield the highest R² value. Consequently, each partition or split of the tree illustrates the most significant factor affecting the metric at that split. Ultimately, this communicates to stakeholders which parameter or combinations of parameters result in desirable or undesirable outcomes. While quantifying and examining the values from a statistical aspect is useful in academia, the uniqueness of the scenario requires an operational interpretation to add value for decision makers. As such, the generated prediction profilers and partition trees are examined through statistical, operational, and modeling lenses. This method of examination provides an operational context for the results and explains the validity of the data.

3.1. UAV results

The prediction profiler in Figure 4 delineates how strongly the individual UAV design factors of altitude, sortie size, and time between launches affect the response variable (i.e., HVU detection). Steeper slopes are indicative of a stronger effect on the response variable. Typically, positive slopes indicate that increasing a given parameter increases the associated response metric, while a negative slope means that increasing the parameter decreases the response metric. A zero or near-zero slope indicates that the parameter has a marginal effect on the metric.

From the prediction profiler, two conclusions can be ascertained: while altitude is the design factor that has the most impact in driving HVU detection, the associated t-values for interaction terms between altitude and number of UAVs, as well as interaction between altitude and time between launch, require that all three factors be studied further.

Figure 5 shows the results of the experiment for UAV ability to detect HVUs in a partition tree. Starting with the three design factors across all 27 data points, the average detections go from about 29 to 289 HVU detections just by changing the flight altitude from 60,000 or 35,000 feet to 10,000 feet. In other words, the middle and high altitudes show similar performance degradation in the modeled sensor.

The next partition shows that the number of UAVs flown is the second most significant factor in HVU detections, but quantity only has a significant impact at 10,000-foot flight altitudes, suggesting that the sensor package is resolution-limited in the model; adding more "eyes" at higher altitudes has minimal effect.

Thus, from the partition tree two conclusions are drawn. Firstly, HVU detection is maximized by flying at 10,000 feet while employing three UAVs per directed search area for a total of 69 UAVs. Secondly, if it is a requirement to fly at high altitude, then simultaneous UAV employment provides a slight improvement versus staggered launch times. Within the model, adding any number of UAVs to the scenario has a positive impact on building search area along the Sonoran coast; however, the value-added diminishes at higher mission altitudes. In a real-world context, this could communicate to force planners that the viability of high-altitude reconnaissance aircraft is contingent on first enhancing sensor resolution for higher altitude flights.

While sortie size and altitude are primary drivers in determining HVU detections, time between launches also affects enemy detections. As a greater number of UAVs deploy, truncating time between launches increases HVU detections. Conversely, staggering times with a small contingent of UAVs results in a slight decrease in HVU detections. In an operational context, the relationship between the quantity of UAVs and launch times suggest that small swarms of UAVs are more effective in an ISR role when launched near-simultaneously. Therefore, if it is economically infeasible to procure a vast squadron of reconnaissance UAVs, the model suggests that engineering the capability for faster launch times would maximize HVU detections.

Overall, these trends make sense in a real-world perspective. Given similar sensors, increasing the distance (i.e., altitude) from the sensor to the target will result in lower HVU resolution; the model captures this trend. Moreover, assuming sensor resolution is not a limiting factor, the expected result is that a greater number of sensors employed should result in higher HVU detections; this trend is illustrated in the bottom-right partition in Figure 5. For a more detailed discussion of UAV context, results, and analysis, the interested reader is directed to Langreck.¹³

3.2. UUV results

Figure 6 shows the effects of the system attributes (UUV composition, speed, and sonar type) on the detection of Sonoran naval units. At an alpha level of 0.10, two variables affect detection: UUV composition and sonar type. However, speed, alone, has little influence on the measure. The prediction profiler further explores the effects when varying these factors. Statistically, there is a drastic decrease in detection when UUVs are using passive sonar only. This makes sense from an operational perspective, as active sonar radiates noise that will pinpoint a vessel of interest. The behavior of speed also makes sense operationally. Increasing the speed of the UUVs equates to more generation of noise; as a result, this noise interferes with the signal-to-noise ratio, making it difficult for the UUV sensors to detect the vessels of interest. The anomaly is the UUV fleet composition. More UUVs should increase detection rates, but the profiler shows a decline after fleet composition of 12 UUVs. This will be further explored using the partition tree.

The simulation results for the UUV experiment are shown in Figure 7, which identifies factor combinations that increase or reduce detection. The initial split exists with the deployment of UUVs (UUV composition). The UUV composition of 8 and 12 UUVs significantly detects more naval units than the 16-UUV fleet. The least-preferred design point occurs with 16 UUVs at speeds of 5 knots. This design point encompasses both active and passive sonar, meaning that there are no significant differences between the detection rates of active and passive sonar in this design. Operationally, the UUVs are traveling too slowly to enter the detection ranges. Furthermore, using active sonar reveals the location of the UUVs, permitting the Sonoran naval units to escape before swarming can occur. The influence of sonar type on detection is most notable between the 8 and 12 UUV compositions. The final split occurs with speed. UUVs traveling at 8 knots detect more units compared with UUVs traveling at 5 or 12 knots. These observations identify that the preferred UUV attributes for detection is a group of 8–12 UUVs traversing at 8 knots with active sonar engaged.

For a more detailed discussion of UUV context, results, and analysis, the interested reader is directed to Wong.¹⁴

4. Conclusions and future work

This article describes the feasibility of using an ACAW as a capabilities developer by creating and testing advanced UAVs and UUVs in support of reconnaissance requirements. Generally, results from the model correlate to what would be expected operationally, validating the application of the ACAW as a force planning tool. Using the JTLS-GO modeling environment and a ready-made scenario for CG18, the study team developed a CONOP for advanced UAV and UUV prototypes in a conflict environment. The next step involved the selection of three factors (attributes) and MOEs. This step permitted validation and verification of the prototypes against the reconnaissance requirements in the scenario. The subsequent procedure involved experimentation in discrete value levels for each factor. Adapting the events from CG18, an automated JTLS-GO wargame produced data for analysis using prediction profiles, partition trees, and course of action analysis.

The ACAW, as designed, can rigorously test capability requirements to find COAs such as CADP. Part of the CADP and COA development requires wargaming. Consequently, the process of wargaming requires the development of databases and scenarios. The JTLS-GO modeling environment has a mature database and is currently used in various multi-national exercises. Adopting the various scenarios and transforming these exercises into an automated wargaming scenario eliminates manning requirements. The automation of wargaming provides multiple, repetitive simulation iterations, giving more confidence to COA selection. Further work in this proposed process includes testing force requirements and examining the results in support of COA selection. Future work along these lines is intended to be undertaken on behalf of the US Navy under the technical supervision of the Pacific Warfighting Center. The ACAW is not limited to wargaming. It can be a tool used to test and explore future capabilities and advanced architecture without the need to wait for operational or stakeholder needs. Most importantly, the ACAW enables credible statistical analysis from experimental outcomes that produce data with a pedigree that can withstand scrutiny.

Footnotes

Acknowledgements

The authors would like to thank R&A, Inc. for their kind technical assistance and permission to use JTLS-GO free of charge. Also, thank you to the Pacific Warfighting Center for inviting the authors to observe CG18 and for sharing the exercise database. A portion of this work was used to meet thesis requirements for students at the Naval Postgraduate School (NPS).

Funding

This work was supported through the Naval Research Program under the direction of Rod Abbot and oversight of the OPNAV N1 in Washington, DC.

ORCID iD

William Hatch

Author biographies

John Langreck graduated from the NPS in 2018 with a Master’s degree in Systems Engineering. He is a Surface Warfare Officer in the US Navy. He is currently the attending Department Head training at the Surface Warfare Officers School (SWOS), Newport, Rhode Island, USA, prior to reporting to USS Milius (DDG-69) in Japan.

Hermann Wong graduated from the NPS in 2018 with a master’s degree in Systems Engineering. He is an Engineering Duty Officer in the US Navy. He is currently stationed at the Southwest Regional Maintenance Center (SWRMC) in San Diego, California, USA.

Alejandro Hernandez is an Associate Professor in the Systems Engineering Department and Deputy Director for the Simulation Experiments and Efficient Designs (SEED) Center at the NPS in Monterey, California. He holds a BS in Civil Engineering from the US Military Academy, an MS and PhD in Operations Research from the NPS, and a Master’s in Strategic Studies from the Army War College. He is a retired Army officer whose assignments include Director of Analysis & Assessment – Iraq and Chief of the Warfighting Analysis Division in the Department of the Army Programs and Resources Directorate. He applies M&S, experimentation, and scenario methodologies to develop robust decision support systems that improve the design, development, operation, and management of complex systems.

Stephen Upton is a Research Associate for the SEED Center for Data Farming at the NPS, where he supports projects that advance the development and use of simulation experiments and efficient designs. He has a BS in Physics from the University of Idaho and an MS in Operations Research and Physics from the NPS. He gained valuable experience as an Applied Scientist in Systems Engineering at George Washington University. He has worked as a research scientist for Referentia Systems, the MITRE Corporation, and the Los Alamos National Laboratory, and served 24 years on active duty with the US Marine Corps.

Mary McDonald is a Research Associate for the SEED Center for Data Farming at the NPS, where she supports projects that advance the development and use of simulation experiments and efficient designs. She has a BA in Mathematics from Northwestern University and an MS in Applied Mathematics from the NPS. She has over 20 years of experience in military simulation and analysis, teaches probability and statistics, and provides guest lectures for modules on combat modeling. Her research interests include evolutionary computation, high-dimensional data analysis, and agent-based simulation.

Anthony Pollman is an Assistant Professor in the Systems Engineering Department and the Deputy Director of the Littoral Operations Center at the NPS, Monterey, California. He holds a BS and MS in Nuclear Engineering from Purdue University, a PhD in Mechanical Engineering from the University of Maryland-College Park, and an Executive MBA from the NPS. He teaches courses in system dynamics, sensors, and mathematical modeling. His research interests include process M&S, operational energy, autonomous systems, and radiation detectors. He is retired Marine and veteran of both Iraq and Afghanistan.

William Hatch is a Senior Lecturer in the Graduate School of Business and Public Policy. He holds a BA in Industrial Arts with a teaching credential from San Jose State University and an MS in Management (Manpower, Personnel, Training and Analysis) from the NPS. He has 22 years of military service that includes serving in leadership and staff positions on board frigates, destroyers, cruisers, and carriers. He has been at the NPS for 20 years and teaches courses in human resource management, education and training, and manpower requirements for the Navy and Marine Corps. His research areas are unmanned vehicles, and civilian manpower, personnel, training, and education problems with international and US militaries.

References

Prochazka

Hodicky

. Modelling and simulation as a service and concept development and experimentation. In: Krivanek

(ed) IEEE international conference on military technologies, Brno, Czech Republic, 31 May–2 June 2017; pp. 721–727. Piscataway, NJ: IEEE.

Cayırcı

Ozcakir

Modeling and simulation support to the defense planning process. J Defense Model Simulat 2017; 14: 171–180.

Owens

MT.

Force planning: the crossroads of strategy and the political process. Orbis 2015; 59: 411-37.

Rolands & Associates, LLC. Corporate overview, https://www.rolands.com/wwa/overview.php (n.d., accessed 12 December 2018).

Hernandez

Hatch

Pollman

, et al. Computer experimentation and scenario methodologies to support integration and operations phases of mission engineering and analysis. In: Rabe

Juan

Mustafee

Skoogh

Jain

Johansson

(eds.) INFORMS winter simulation conference, Gothenburg, Sweden, 9–12 December 2018; pp. 3765–3776. Catonsville, MD: INFORMS.

Law

AM.

Simulation modeling. 5th ed. New York: McGraw Hill, 2015.

Defense Acquisition University. Glossary of acquisition terms, https://www.dau.edu/glossary/Pages/Glossary.aspx#!both|M|27925 (accessed 27 August 2019).

Green

JM.

Establishing system measures of effectiveness. Report, Raytheon Naval and Maritime Integrated Systems, San Diego, March 2001.

Cioppa

Lucas

TW.

Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics 2007; 49: 45–55.

10.

Sanchez

PJ.

Very large fractional factorials and central composite designs. ACM Trans Model Comput Simulat 2005; 15: 362–377.

11.

MacCalman

Vieira

Lucas

TW.

Second-order nearly orthogonal Latin hypercubes for exploring stochastic simulations. J Simulat 2017; 11: 137–150.

12.

SAS Institute Inc. Partition models, https://www.jmp.com/support/help/14/partition-models.shtml (12 July 2018, accessed 27 September 2018).

13.

Langreck

A feasibility study using scenario methodologies on future unmanned aerial system capabilities. MS Thesis, Naval Postgraduate School, USA, 2018.

14.

Wong

Evaluation of future unmanned underwater vehicle capabilities in an automated computer-aided wargame. MS Thesis, Naval Postgraduate School, USA, 2018.