Abstract
A critical component of the military war game planning process is selecting who should participate, as these participants heavily influence war game outcomes. These outcomes directly impact both strategic and operational decision-making and defense planning, shaping both future defense policy and budget. In this paper, we propose a novel team selection algorithm and decision-support tool combining methods from multiple criteria decision analysis and network analytics to select and visualize a group of war game participants. This method accounts for the diverse requirements of the decision-maker. The results are not only applicable to war games, but also to any team selection domain, such as employee hiring and college admissions.
Keywords
1. Introduction
Unified Quest is a year-long series of studies, workshops, and war games focused on examining the future challenges, issues, and potential solutions for the defense of the United States. 1 The program was originally established by the US Army Training and Doctrine Command (TRADOC) in 2003 as a part of the Army’s Campaign of Learning to help anticipate post-Cold War threats. 2 Currently, the US Army Capabilities Integration Center (ARCIC) runs a series of studies and exercises that build off of one another and culminate with a week-long war game that integrates all the research and results from previous Unified Quest events. The outcomes from these war games have a direct impact on both strategic and operational decision-making within the Army, shaping both policy and budget. For example, Unified Quest 2012 - Building Partners and Partner Capacity participants identified the need of the Army to develop regional expertise, cultural awareness, and language skills. 3 Traditionally, the Army has been viewed as a global response force able to rapidly deploy anywhere at any time. The recommendations derived from this war game helped influence the Army’s concept of “regionally aligned forces.” Units are now assigned a specific region of the world on which they become subject matter experts, and ideally conduct training exercises within the respective host nations. This transition is still a work in progress, but exemplifies the impact that Unified Quest has on the future of the Army.
The focus of Unified Quest shifts annually. Topics from previous years include conventional warfare in a multi-national environment, cyber warfare, air-sea battle concepts, and setting up communication networks in foreign countries. Unified Quest 2013 examined how an operational working group would conduct force entry operations against a complex mix of state and non-state adversary forces in a predicted 2030 environment. 2 Two groups, one equipped with current capabilities and the other with potential future capabilities, were presented with the same scenario in order to identify the opportunities and gaps that resulted from the difference in approaches, technologies, and capabilities. The outcomes from each of the Unified Quest war games have helped shape defense policy for over the last decade. ARCIC plans on running Unified Quest 2015, but the specific details of the war game are still under development.
1.1 Motivation and purpose
A key component of the military war game planning process is deciding who should participate, as these participants heavily influence war game outcomes. The ideas generated by the selected participants of Unified Quest 2015 will become defense policy recommendations provided to senior military leaders and policy-makers. In this paper, we describe a decision-support tool, NetSel, that combines methods from multiple criteria decision analysis and network science to select and visualize a group of war game participants that meet the needs and requirements of ARCIC for Unified Quest 2015.
With any military war game there is an established focus or goal; however, the focus changes from war game to war game. Therefore, it is important that the participants be well-versed in the subject at-hand and have the ability to make significant contributions to the conversation. Without the right group of participants, the results of the war game may not accurately reflect the true needs of the Army in 2030 and beyond. We apply the primary objective of Unified Quest 2015 to identify types of participants who are best suited to the war game and, ultimately, could contribute most. For example, if the war game were focused on the future of Chemical Warfare, then ARCIC may want to include chemists, scientists, Army Chemical officers, experts in decontamination, hazardous materials handling experts, and other various professionals that would be able to provide insight into chemical warfare. Each of these subject matter experts has specific knowledge, education, and experiences that can be used to categorize their participant type. Sets of desired skills can be a guide for the creation of a list of “ideal” individuals who we would like to participate in Unified Quest. By examining similarities between a potential participant and an “ideal individual,” we can begin to identify who should be selected for the war game. This concept is known as team selection.
2. Literature review
The concept of team selection is used constantly in aspects of everyday life. Team selection can be used for various purposes, from selecting board members for a large company, picking applicants for college admissions, and choosing a project team for a business endeavor, to drafting players for a sports team. For example, Stein 4 used network analysis to identify candidates for knowledge acquisition in the domain of management information systems. Similar applications were investigated by Medsker et al. 5 and Hamilton and Breslawski. 6 In every team selection problem, the group’s success is highly dependent upon the constituent members. A decision-maker needs to properly select a group that will meet a set of goals. The issue is that every situation is unique and will have its own distinct set of objectives to accomplish. Based on these unique goals, a decision-maker must develop a method to select a group that will effectively meet those objectives. As a result, numerous methods have been used to select groups for different situations.
This paper constitutes the first multi-criteria, network analytic approach for solving team selection problems, as it incorporates both ranking/selection and visualization methods as desired by the decision-maker. Potential group members and their multiple attributes are represented by nodes, and the possession of a particular attribute is signified by a link. We apply adapted notions of structural equivalence, first developed by Lorrain and White, 7 to select individuals with desirable skill-sets. We describe various definitions of structural equivalence found in the literature, as well as our adaptation. This paper makes the following contributions to the existing team selection literature: (1) a multi-criteria framework to analyze the synergistic effects of skill-sets rather than individual skills in isolation; (2) inclusion of stakeholder preferences for multiple individuals possessing particular skill-sets rather than separate skills; (3) a metric that assesses the overall quality of a potential group; and (4) an interactive decision-support tool with network visualization usable by real-world stakeholders without a technical or algorithmic background.
Datta et al. 8 investigate the value of social connections when selecting individuals for a software engineer team. The collaboration network analyzed was built using data from Github, the social coding platform. In the model, users are represented by nodes and have a set of skills, for example programming languages, that a project could demand. Users are also constrained by their capacities, which correspond to the maximum number of tasks that they can be assigned. Of course, a user can only be assigned a task if he or she possesses the corresponding skill. A project constitutes a subset of all possible skills available. Every pair of users in the network is said to have a “collaboration cost,” which is a weighted proxy representing the likelihood and ease of future collaboration. This cost is directly proportional to the number of projects two individuals have worked on together in the past. In other words, if two individuals have worked on numerous projects together, their associated collaboration cost is small. In this work, optimizing a team equates to minimizing overall collaboration cost. This optimization problem is shown to be NP-hard, so the authors design approximation algorithms. (NP-hard refers to the set of decision problems for which there exists a polynomial-time reduction to a problem in the class NP. The class NP is the set of decision problems (yes/no answer) for which a particular solution can be verified in polynomial-time, although not actually solved in polynomial-time outright.)
However, collaboration cost does not readily generalize to non-adjacent nodes and the network as a whole, an inherent methodological problem acknowledged by the authors. Effective teams must work well together, insomuch as the individuals must have complimentary personalities, similar work ethics, etc. In addition, collaboration cost is an artificial construct that does not take any such social aspects into account. Past collaborators do often work together in the future, but not always. For example, suppose a project failed, took more time than expected, or angered several clients. Even if the team has worked together many times before, in this case it is unlikely that they would work together again. New users, for example recent college graduates, have high collaboration costs with respect to all other users in the network, even if their social networks connect them to many. This observation leads us to believe that true social networks are essential when modeling collaboration likelihoods, since they provide insight into social, rather than work, connections, as well as frequencies and durations of contact during collaboration.
Guttman and Shpitalni 9 apply integer programming formulations to address team selection problems. The objective function reflects the need to build an effective team given the skills and expertise demanded by a project, as well as the set of individuals possessing them. Several specific objective functions are considered, which attempt to represent a range of possible goals a human resource manager might have in mind: (1) minimize team size, and (2) minimize team cost. The authors demonstrate that the first goal is similar to the set-covering problem, since the aim is to produce a minimum size team such that all expertise areas are “covered.” Variations of the first goal impose additional constraints that force or prevent certain combinations of individuals. For example, an individual might only contract to a project if his partner joins him. Alternatively, individuals working separate aspects of a project might not be permitted to exchange confidential data. Another variant allows for stakeholders to demand skills in a differential manner, although this extension is limited in the ways discussed above. Since integer programs are not polynomial-time solvable in general, the authors explicitly solve a small example consisting of five individuals and six areas of expertise to optimality, and use approximation heuristics for larger problems. Specifically, the authors compare three genetic algorithm approaches and argue for the superiority of one referred to as “plain vanilla.”
Guttman and Shpitalni 9 present multiple minimization objectives. The modeling framework could be expanded to a multi-objective formulation, as many real-world team selection problems require stakeholders to consider these conflicting objectives simultaneously. A simple first step could be a multi-objective function that weights each constituent objective. Next, social connections and collaborative team externalities are not taken into account. In reality, a team is more than the sum of its parts, or members. The objective functions do not reflect team efficiencies gained through social experiences, personalities, etc. Complimentary skills are not rewarded in any objective function. While our methodology does reward an individual for having a “desirable mix” of attributes, our framework does not express it in the form of an objective function or formal optimization setting.
Baykasoglu et al. 10 consider team selection problems under a deadline using fuzzy optimization methods. Triangle fuzzy numbers and linguistic variables are used to evaluate the skills of each potential team member with respect to the skill requirements demanded by a project. Four core skills are considered: oral communication, technical expertise, problem-solving, and decision-making. The fuzzy approach put forth allows stakeholders to assess a candidate’s suitability in the face of uncertain, unreliable, or objective qualification information. The fuzzy optimization model takes the following factors into account: the number of candidates to be selected, candidate suitability, candidate salary requirements with respect to a project budget, and project deadlines. Suitability and team-size objectives are discussed, and the model is solved using a simulated annealing heuristic.
In both Datta et al. 8 and Guttman and Shpitalni, 9 the authors discuss extensions that include demand for multiple “units” of a skill, such as Java programming. However, it is neither sensible nor realistic to demand “3 Javas,” which is equivalent to demanding a “super worker” with effectively infinite capacity and many skills. We contend that this “super worker” is suboptimal, since he or she cannot offer the benefits of collaboration with others. Instead of demanding “3 Javas,” a project would realistically demand three individuals competent in Java. Alternatively, “3 Javas” could reflect an expertise level in Java. Extensions of our work could account for expertise level in each area under consideration. In addition, the requirements of the exact tasks, skills, and time are often not known before it commences. In these cases, it is important to build a team with the appropriate mix of skills to attack the general problem-domain of the project.
Our framework, NetSel, provides stakeholders with the ability to demand not only multiple individuals possessing a particular attribute, but even multiple individuals of a specific profile, that is, a set of attributes and experiences. This approach expands upon the methods described by Guttman and Shpitalni 9 to reflect synergistic effects of entire skill-sets. Baykasoglu et al. 10 are concerned with maximizing the minimum suitability of a team member, but neglect overall team suitability for the project. NetSel accounts for team suitability with a “group optimality score,” which acts as a proxy for overall team quality. Under a fuzzy optimization approach, it seems that one could adapt the objective function by using an average suitability score.
Adapted structural equivalence methods allow us to expand upon current team selection algorithms as described above. Structural equivalence is a way of measuring the similarity of connections amongst nodes: it is a concept of node substitutability with respect to relational ties. Lorrain and White 7 define two nodes to be structurally equivalent if they have all connections, that is, to other nodes, in common. Structural equivalence is a commutative relation, since equivalent nodes A and B indicate that A is structurally equivalent to B, and B is structurally equivalent to A. The concept was conceived in the social network analysis community, although it readily extends to networks at large. The discussion in Sailer 11 illustrates the limited utility of such a strict definition, since two people rarely have exactly the same connections. Sailer 11 suggests that the definition can be generalized and relaxed. The most notable modification put forth is to reduce the number of social relations considered, thereby loosening the number of requirements imposed to declare structural equivalence. Sailer 11 discusses the use and caveats surrounding algorithms to group nodes and reduce the number of social relations.
In effect, these weakened definitions of structural equivalence are still limited, since the quality of equivalence remains binary. Any of the classic definitions provided still imply that two nodes are structurally equivalent or not, with nothing in between. These binary characterizations prevent researchers from directly comparing nodes in terms of their substitutability, as well as their comparisons to a particular “benchmark node.” In other words, if nodes A and B possess nearly identical connections, but differ by one or two, we would simply declare that A and B are not equivalent. We adapt traditional notions of structural equivalence to define it as a continuous quantity. This modified definition, while simple, allows researchers to answer questions such as (1) How “equivalent” are two nodes in a network?, and (2) Which two nodes are more substitutable: A and B, or A and C?
Suppose we wish to determine the degree to which nodes A and B are equivalent. We must first designate a “benchmark node” and then count the proportion of connections the other node has in common. Without loss of generality, assume A is the benchmark node. The “structural equivalence score” of B with respect to A is then the following fraction:
3. Methodology
In this paper, we develop and implement a multi-criteria network analytic decision-support model for war game participant selection and visualization. The potential participants and “ideal participants” will be included in the network, but instead of examining social relationships between participants, we consider a person’s ties to their attributes. An attribute is a characteristic of an individual that can include areas of professional expertise, academic background, military experience, and languages spoken. In our network model, each person is connected to his or her own attributes, which helps define them for the purposes of team selection. This creates a bipartite network (two-mode matrix), which has two types of nodes: people and attributes. In addition, the “ideal participants” are constructed using a set of attributes based on the needs of the war game. With a complete set of potential participants and “ideal participants,” we create a network visualization that displays both the people and their attributes. A simplified example of one potential participant and one “ideal participant” with their respective attributes is shown in Figure 1.
Ideal 1 and Person 1 with their respective attributes.
We can now observe the relationships that are present between the potential participant and “ideal participant” with their respective attributes. Person 1 possesses six attributes and Ideal 1 possesses eight attributes. Network science methods allow us to easily compare the attributes of the potential participants and the attributes of the “ideal participants.” We can see in the network visualization, presented in Figure 1, that Person 1 shares four attributes with Ideal 1: Academia, History, Mathematics, and Military. The more attributes that a potential participant shares with the “ideal participant,” the more likely they will be selected for the war game. This is the premise of our participant scoring process. To compare the potential participants and the “ideal participants” on a larger scale, we use a method called structural equivalence.
3.1 Data collection
In order to construct this network, we collect data on each of the potential participants. Pertinent information collected in the context of Unified Quest consists of academic background, professional sector, military experience, and languages spoken. Academic background refers to an individual’s bachelor, master, and doctoral fields of study. Professional sector refers to organizational affiliations of the participant. Military experience describes the branch of service and specialties within the military, if applicable. Finally, languages spoken express an individual’s conversational language ability in various languages from around the world. These four categories of characteristics form an individual’s attributes. We collect information on these four attributes via a four question survey:
For what fields of study have you earned a degree?
How would you categorize the sector of all organizations that you have been a member of over the past 10 years?
If you are currently a member of a military organization or if you had prior military service, what are/were your specialties?
What languages do you speak at a conversational level?
Each question has a list of preset responses such that the potential participant can select all that apply. The survey format allows for flexibility of responses while ensuring uniformity in our dataset. By allowing the participants to “check all that apply,” we do not limit the number of attributes that they can select. This helps us capture all the attributes that the participants believe they possess, rather than forcing them to pick only their top few attributes. Further, this format allows us to accurately represent the participants in our analysis; it should be noted, however, that this has the potential limitation of introducing self-bias. Nevertheless, the preset responses in the survey prevent two people from using different wording to describe the same attribute and eliminate the possibility of misspellings. The preset responses ensure that individuals are characterized by attributes that will be examined during our analysis, rather than collecting information that is not considered for analysis in our methodology. In addition, the preset responses provide us with a good estimate of the size of the dataset. If there are n participants and m attributes, then we know the dataset will contain n × m entries.
In this paper, we demonstrate our multi-criteria network analytic decision-support tool for war game participant selection using notional data from a past war game. The only data that were collected were participant names and their occupational organizations. This raw dataset contributes little to our structural equivalence analysis, but we were able to use it as a starting point. Specifically, we expanded the raw dataset to find more information about each individual using open-source internet searches. Following this, we coded the data to reflect the following 18 attributes considered in this study: academia, Arabic, communication, engineering, mechanical engineering, systems/industrial engineering, mathematics, government, history, infantry, intelligence, military intelligence, law, political science, politics, public administration, research, and signal.
3.2 Constructing the network
The survey data is currently formatted with an individual’s name and a list of only the attributes they possess. Using social network analysis software, we construct a person-by-attribute (two-mode) binary matrix. The matrix Z = (zij) has dimension n × m, where n is the number of people and m is the total number of attributes from the survey. In this case, m = 18. For any matrix entry, zij equals 1 if person i possesses attribute j, or 0 otherwise. The matrix representation of the participants’ attribute data allows us to develop a network diagram to better visualize the connections that exist within our dataset. The network is bimodal, meaning that it contains two classes of nodes. Participants are represented by red circles and attributes are represented by green squares. A link (i,j), which exists between person i and attribute j, in the network corresponds to a matrix entry zij = 1. Only person–attribute links exist, rather than person–person or attribute–attribute links. The simplified two-mode network displayed in Figure 2 shows all the participants from the previous war game dataset with only several important attributes being displayed.
Network visualization of the war game dataset.
3.3 Archetypes
In addition to the collection of the candidate participant data, we created a set of “ideal individuals” for the war game. The concept is to construct a small, fictional set of participants, each of whom play an important role in the war game, who constitute an optimal benchmark against which we can compare Unified Quest applicants. We refer to these “ideal” individuals as archetypes. An archetype is a plausible set of attributes that, taken together, describe an “ideal individual” for a given war game. An archetype represents the stakeholders’ beliefs that an individual possessing this set of attributes in combination will be able to generate productive contributions to the war game. Each of the archetypes is designed to fill a certain role or niche within the war game. For example, in a war game looking at biological warfare, we would have an archetype named “Scientist” who would likely possess the attributes chemistry, physics, and research. The idea is that this single individual will more effectively contribute to the war game by the synergistic effects of his or her constituent attributes. This is preferable to three individuals who each possess one of these attributes because they are limited to drawing only on specific, targeted experiences when solving a problem, rather than a holistic perspective derived from multiple types of experiences. The key for selecting the attributes for each archetype is looking at what blend of attributes in a single person would provide the most value to the war game.
We elicit the stakeholder preferences to determine which attributes are important to the war game. We use the Delphi idea generation technique to establish the stakeholders’ preferences, where each stakeholder anonymously submits his or her preferences of important attributes. This eliminates several biases from the archetype generation process. Next, the stakeholders determine the total number of distinct archetypes desired for the war game. Our analytic process for participant selection can consider up to five distinct archetypes; therefore, the stakeholder must select between one and five archetypes. Another noteworthy characteristic of the archetypes is that they can share attributes (i.e. the archetypes are not necessarily mutually exclusive). The stakeholders also determine the number of each archetype they would like in the war game.
3.4 Individual comparisons
By applying stakeholder preferences we can construct a network of archetypes and attributes. This “ideal network” can then be used as a baseline to evaluate potential participants. The underlying principle for the structural equivalence analysis is how well an individual’s attributes match the attributes of an archetype. Figure 3 demonstrates a simplified example of the relationship between participants and archetypes with their respective attributes.
Network comparison of Person 1 to Archetype 1 via person-by-attribute construct.
The network visualizes the attributes possessed by the participant and archetype. The score for a participant with respect to an archetype is calculated by counting the number of attributes shared by the participant and archetype, then dividing that by the total number of attributes the archetype possesses. This is the premise for how we evaluate individuals for selection for the Unified Quest war game. This approach of counting shared attributes by visual inspection of the network diagram is sufficient for a small example, but becomes impractical for larger datasets. Therefore, we utilize the matrix form of the network to perform structural equivalence analysis on the dataset.
The concept of structural equivalence analysis is the same as counting the common attributes, but uses matrices rather than graph representations. At this point, we have discussed two different types of matrices: a Person-by-Attribute matrix and an Archetype-by-Attribute matrix. Structural equivalence allows us to compare these two different datasets (two-mode matrices). The goal of the analysis is to calculate a score for how well each participant matches each archetype. To do this, we evaluate the entire participant list against each archetype, successively. Figure 4 displays an example matrix representation of the network from Figure 3, which demonstrates Person 1’s score with respect to Archetype 1.
Structure equivalence score: Person 1 with respect to Archetype 1.
Along the top of Figure 4, each column corresponds to 1 to m attributes that are under consideration, in this example. The first row corresponds to Archetype 1, while the second row corresponds to Person 1. The attributes that each possess have a “1” in the corresponding attribute column. The final row is the Equivalence row, which determines if the archetype and participant share a common attribute. If both Archetype 1 and Person 1 have a value of “1” in an attribute column, then the Equivalence row will return a value of “1.”
To better explain how the score for a person with respect to an archetype is calculated, we will use the simplified mathematical notion described below. As mentioned, the first row corresponds to Archetype 1. For each matrix entry, aj, in row 1:
The second row represents Person 1 and for any matrix entry, bj, the same binary logic applies as it did in row 1.
The data for rows 1 and 2 comes from the stakeholder and participant surveys, respectively. Lastly, the third row calculates the structural equivalence between the two. Equivalence exists if both the Archetype and Person have an attribute in common. For any matrix entry, cj, in the equivalence row:
The equivalence is calculated for every attribute j. We next find the structural equivalence score for participant i with respect to archetype k, xik:
For this equation we sum the values in the equivalence row, and then divide by the total number of attributes that archetype k possesses. In the example, Person 1 and Archetype 1 share three out of Archetype 1’s seven attributes, giving Person 1 a score of 42.9%. This example shows a single person in comparison to a single archetype. This same calculation is performed for every participant. Therefore, we would have a list of scores for all individuals in regards to Archetype 1. The process is repeated for each of the archetypes. After all the scores have been calculated, we can determine the archetypes each individual best fits with. We create an ordered list of archetypes that display an individual’s best-fitting archetype to worst-fitting archetype. In some cases, a participant may have the same score for two or more archetypes. If this happens, we use the archetype-based tie-breaking rule; this rule states that we will place the archetype with the smallest index highest on the ordered list.
3.5 Sorting score, rank, and effective rank
Every participant has a score for each archetype and an ordered list of highest scoring archetypes. Note that it is quite possible that two or more people will have the same score with respect to a particular archetype. To address this concern, we use a participant-based tie-breaking rule that states we will select the participant who is earlier on the list of participants. The order of a participant on the list may appear arbitrary, because it could merely be alphabetical or based on time of registration. Therefore, we justify our participant-based tie-breaking rule by ordering the participants based on a sorting score, which uses the L2-metric (Euclidean distance) to determine how close a participant is to a “perfect individual” (a person who possess every attribute for each archetype). The lower a participant’s sorting score, the closer the person is to the ideal individual.
The sorting score creates an additional, unique score for every participant, since no two participants are likely to have the same scores for all the archetypes. This score determines the order of people on the participant list; the lowest score being on top of the list and the highest score at the bottom. Having the participants ordered by sorting score, rather than an arbitrary means provides justification for our participant-based tie-breaking rule.
The rank of a participant is how his or her score for a single archetype compares to the scores of other participants within that same archetype. This is useful because some archetypes are more competitive than others. The high score for one archetype could be 96%, while for another the high score could only be 73%. This means that a participant’s highest scoring archetype may not yield that individual’s best rank. Rather, the participant’s lower scoring archetype may produce a higher rank due to the fact that the rest of the participants scored lower for that archetype as well. Therefore, it is important that we assign each individual a rank with respect to each archetype. Thus, the participant with the highest score ranks first, the person with the next highest score ranks second, and so on. Suppose two individuals have the same score for an archetype, then, generally, the two participants would share the same rank. This causes our selection method to become ineffective, showing the utility of sorting score. If two people have the same archetype score, then the individual with the lowest sorting score will receive the lower rank, where lower means better.
Effective rank examines how an individual’s rank for an archetype will change given that people previously ranked lower (lower being better) than that individual are selected for a different archetype. Even though an individual may not initially rank highly enough for an archetype to participate in the war game, they may still be asked to participate if others previously ranked above him or her are selected for different archetypes. Thus, effective rank equals an individual’s actual rank minusgg the number of lower ranked participants who were selected for a different archetype. Using the effective rank of each participant, participant selection is a row-by-row, iterative process that considers the results of the participant selections physically above an individual on the ordered list. Although we can select individuals who meet the stakeholder requirements of each archetype, we develop an optimality metric for the overall quality of the group selected for participation in the war game.
3.6 Group optimality
To address the issue of the overall quality of the participant group, we develop a group optimality score. Let xik be the score of participant i for archetype k, given that participant i was selected for the war game to fill a slot for archetype k. Let p be the total number of participants in the group selected for the war game. Then,
The group optimality provides us with a score between 0 and 100%. A score of 0% represents a group of participants that does not share any attributes with any of the archetypes, whereas a score of 100% represents a group of participants that perfectly satisfy all the archetype requirements that the stakeholders have defined. Given the wide variety of attributes and archetype possibilities, achieving a score near either 0% or 100% may indicate an issue with how the archetypes were defined. This could possibly signify that the archetypes were either too specifically or too broadly defined. Therefore, we use this metric to gain quick insights into the effectiveness of the multi-criteria selection process. This will also be useful when performing a sensitivity analysis, which depicts how changes in our participant dataset, archetype preferences, and analysis affect the overall group score.
3.7 Decision-support tool
Our decision-support tool, NetSel, applies the adapted structural equivalence algorithm described above to select and visualize the best group of participants for a particular war game. NetSel is a Microsoft Excel-based, user-friendly tool that requires no technical background on the part of the user. NetSel also makes use of Microsoft Visual Basic for Applications (VBA) as well as NodeXL, a network analysis Excel add-in. Microsoft Excel’s extensive user-base makes it a natural platform on which to build our decision-support tool, allowing nearly any decision-maker to seamlessly run his/her analysis. In the interest of brevity, we omit further discussion of the decision-support tool.
4. Results and discussion
We now compare the selections resulting from sample preferences selected in the algorithm to classic network analysis metrics. Typical centrality measures, widely used in the network science community, aim to determine the most influential individuals in a social system, based on their connections and potential information flow throughout the network. Individuals, represented as nodes, are connected through their attributes. For example, if two individuals studied engineering then they are connected by an edge. This network is obtained by performing a data fold, which is defined as projecting two-mode data (bipartite network) to a single mode. 12 This technique transforms the two-mode (person × attribute) matrix to a one-mode (person × person) matrix by post-multiplying the two-mode matrix by its transpose. Since we are not concerned with tie-strength in this work, we coerce all link weights to 1 after the data folding transformation.
Four normalized classic network centrality metrics serve as quantitative proxies for individual influence in this social system. Degree centrality represents the connectedness of an individual, where the degree of a node refers to the number of edges incident upon a node. 13 If an individual is connected to many, that individual is said to have high degree centrality. Closeness centrality, which refers to how connected a node is to its neighboring nodes, quantifies an individual’s access to all others in the network.14,15 This metric represents the inverse of the average distance that each node is from all other nodes in the network. Thus, the more central a node is, the closer it is to all other nodes in the network. If an individual is many links away from numerous others, that individual is said to have low closeness centrality. Eigenvector centrality measures the influence of each node in the network by assessing how well-connected a node is to the parts of the network with the greatest connectivity. 13 This represents an individual’s connectedness to prominent or influential individuals. Under this metric, an individual is classified as important if he knows important people. Finally, betweenness centrality measures the number of shortest paths from all nodes to all others that traverse through that node of interest. 13 In other words, betweenness centrality quantifies the number of times a node acts as a bridge along the shortest path between two different nodes. High betweenness nodes are often found at the intersections of more densely connect network communities, making them very influential in the network with respect to potential information flow. If a particular individual is the only one connecting two large clusters, that individual is said to have high betweenness centrality.14,15
We compare our participant selections using NetSel to the most influential nodes of the previous war game social network (not displayed here due to sensitivity). We find that many of NetSel’s selections agree with the aforementioned centrality metrics, as shown in Table 1.
List of selected participants, using NetSel, and their rank for each of the four network centrality metrics. Only individuals in the top 10 for each metric are shown here. A cell reads “N/A” if that individual is not ranked in the top 10 for that metric.
Table 1 shows that NetSel’s selection algorithm chooses individuals who would be highly influential in the full participant system. Further, these two selections made by NetSel are frequently important from a network analytic perspective, as Mr W and Dr X are ranked in the top 3 for 63.04% and 54.35% of centrality measures computed, respectively. Note that selection-centrality agreement depends on stakeholder preferences. Participants are selected based on archetype attributes, as well as attribute weightings (i.e. importance of one attribute over another); see Jessop 16 for more details. Once preferences change, participant selections change. Thus, the robustness of selection-centrality remains to be shown. In addition, the value of such agreement depends on the utility of the centrality metrics. If network scientists believe centrality metrics aptly characterize social influence, then selection-centrality relationships provide promising directions for further research. If these relationships are shown to be robust, albeit in varying degrees, then our selection algorithm could serve as a powerful, simultaneous representation of multiple centrality metrics. This multi-criteria aspect could allow researchers to analyze network data more efficiently.
4.1 Model verification and validation
Since the previous war game data was limited, no war game outcome evaluation information was available to use for model validation. As a result, we had no simple way to verify and validate our participant selections, or compare them to actual war game selections made by ARCIC. Instead, we verify our team selection and visualization framework with sample stakeholder preferences in our decision-support tool, NetSel. Specifically, we selected participants for three sample war games. We used the first as a benchmark for comparison. In this case, all attributes were uniformly weighted. In addition, technical and intelligence skills are in greater demand than the others, as shown by the numbers desired for all archetypes, henceforth referred to as “team composition.” The archetype descriptions for this baseline are provided in Table 2.
Description of archetypes along with team composition. The team composition distribution is not uniform, as archetypes 3 and 4 will make up the majority of this team.
We investigate differences in individuals selected in response to changes in stakeholder preferences, another form of sensitivity analysis. Few changes in selected individuals across preferences could indicate homogenous potential participants or stark inequality in participant qualifications. If war game applicants have similar profiles, in terms of attributes, experiences, and qualifications, then changes in preferences would not make an individual “less suitable for selection.” Further, if some individuals possess many attributes under consideration, while others possess few, the former are likely to be selected regardless of preferences. Individuals possessing more, especially diverse, attributes fit a wider range of war game requirements. We would expect the greatest sensitivity when there are similar numbers of attributes across participants and when individuals under consideration are all specialized in their skill-sets, implying participant heterogeneity. For demonstrative purposes, we input preferences for two sample “comparison war games” for this assessment, each differing from the benchmark in one way: non-uniform attribute weightings and the distribution of the archetypes. Each comparison war game is presented in Tables 3 and 4.
Comparison war game 1, with attribute weightings. In this case, technical, language, and intelligence skills are highly valued. This war game features identical archetype definitions and team composition as the benchmark.
Comparison war game 2, with a different team composition compared to the benchmark, but uniform attribute weightings. In other words, stakeholder preferences exhibit a relatively uniform team composition distribution.
Although not displayed here (due to sensitivity), the team selections were nearly identical for all three sample war games. In other words, the same individuals are generally selected for each war game. With respect to the baseline, only three and four individuals differ for the first and second comparison war games, respectively. Here, selection is not sensitive to changes in stakeholder preferences. We explore the previous war game data in an attempt to explain this robust selection behavior. Firstly, we find that potential participants are relatively heterogeneous. There are very few attributes to which a great proportion of individuals connect, as shown in Table 5.
Absolute number and percentage of individuals possessing each attribute under consideration. There are 57 individuals in this dataset. There are only five attributes with high proportions, indicating relative heterogeneity in individuals.
Next, we investigate the number of attributes each individual possesses. We find that on average, participants possess 2.32 attributes, with a standard deviation of 1.17 attributes. Given these figures and that there are 18 attributes in all, we believe that there are no “outlier participants” with many more skills than most.
This previous war game data exploration has not yielded immediately intuitive insights for robust selection. However, this example features only one preference adjustment for each type, attribute weightings and team composition. For the previous war game data, selection may in fact be sensitive. To truly investigate selection robustness, many preferences should be experimentally tested. Efficient determination of selection robustness remains an open question, even empirically. A more powerful extension to this research would derive specific criteria, such as the two described above, in participant data that could predict selection robustness or sensitivity. Such formulations would remove the need for laborious empirical investigations.
4.2 Concluding remarks
This paper contributes a novel team selection algorithm with decision-support tool that is not only applicable to war games, but also any team selection domain, such as employee hiring and college admissions. By construction, this multi-criteria network analytic methodology chooses and visualizes individuals that best align with stakeholder preferences. However, NetSel could be improved to include social interaction effects between team members, an area that remains poorly understood in the team selection literature. This work also provides promising research directions surrounding selection-centrality relationships. Connections between these two approaches could reveal more efficient and detailed network analytic methods, applicable to a myriad of network problems.
Footnotes
Declaration of conflicting interest
The authors declare that there is no conflict of interest.
Funding
The first author is supported in part by the National Science Foundation (NSF) under Grant no. DGE1255832. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF, Pennsylvania State University, US Military Academy, or United States Army.