Abstract
Envisioning new kinds of operations requires systematically developing architectures, work procedures, and artifacts to support human and machine agents in coordinating within dynamic environments. Accurately predicting how envisioned operations will unfold is challenging as (1) early design-phase descriptions of architectures, work procedures, and artifacts are often underspecified, and (2) key outcomes of interest emerge from interactions between cognitive work and environmental dynamics. This paper discusses how computational simulation of work can serve as a discovery tool for envisioning future operations. We introduce a three-phase approach using the Work Models that Compute (WMC) framework, which involves converting paper-based representations of work into computational models, developing scenarios and test conditions, and simulating work dynamics to analyze emergent behaviors. We illustrate this approach through a case study on developing contingency management procedures for envisioned air transport operations, specifically Urban Air Mobility (UAM). The case study demonstrates how computational simulation can (1) reveal the need for clearer design specifications, (2) uncover interactions and emergent behavior that may lead to undesirable outcomes, such as coordination surprises, and (3) identify trade-offs between multiple design options. Insight from simulation can complement other cognitive systems engineering methods to refine and enhance the feasibility and robustness of envisioned operations.
Keywords
Introduction
New data sharing and algorithmic capabilities are envisioned, developed, and deployed to make operations safer, more efficient, and more productive. In addition to improving existing systems, new technological capabilities seek to enable new systems and new forms of work. For example, autonomous capabilities afford new operations at increasing scale—with more interdependencies and increased complexity. A notable example is air transportation, with significant interest in developing and deploying new autonomous aircraft at scale in low-altitude short-range operations, represented in envisioned operational paradigms like Urban Air Mobility (UAM) (FAA, 2023). Part of designing and realizing new operational paradigms is integrating automated and human capabilities to create systems and procedures for human and autonomous actors to coordinate in a dynamic environment.
Operations in dynamic and unstructured environments involve a collective of human and machine agents coordinating and managing dynamic processes, such as a flight crew working with automated systems to manage their aircraft (Hutchins, 1995), air traffic controllers managing multiple aircraft in their airspace (Smith et al., 1999), and mission controllers and astronauts managing systems on board a spacecraft (Watts-Perotti & Woods, 2007). Work demands on human agents in these systems are often characterized by how interactions between dynamic processes in the work environment and the actions of human and machine agents play out over time. For example, the timing for a pilot to extend the flaps and landing gear is determined by the phase of the flight, the position, and the airspeed along the approach path (Hutchins, 1995).
Field studies of real-world operations show that the ability of the collective of human and machine agents to keep pace with the tempo of operations during disruptions is an important determinant of system performance (Woods & Hollnagel, 2006). Failure to keep pace can lead to serious consequences, resulting in coordination breakdowns between people and automation and causing accidents. A challenge when envisioning new operations is making accurate pre-deployment predictions of how interaction effects will play out dynamically (Miller & Feigh, 2019; Woods & Dekker, 2000). New technologies and inventions are often associated with ungrounded predictions about the expected impact of the enhancements. Issues with pace and dynamics of work often only become apparent when prototypes of multiple system elements are integrated and tested for the first time, when it is typically costly to make major revisions to the underlying system architecture. Relative to work dynamics, we define feasibility of an envisioned operational paradigm as the degree to which the temporal behavior of the envisioned system achieves desired outcomes. Similarly, we define robustness as the degree to which the temporal behavior of the system allows effective responses to disturbances in the system. Increasing robustness is equivalent to expanding “the set of disturbances that the system can respond to effectively” (Woods, 2015, p. 2).
Using functional modeling and computational simulation capabilities, this paper shows a technique to evaluate envisioned operations that involve new technology and reveal insights into the emergent effects caused by the interactions between people, technology, and work. While several modeling frameworks exist for model-based evaluation of distributed work systems (Hollnagel, 2012; IJtsma et al., 2019; Leveson, 2011), the use of simulation capabilities can be extended and used as a discovery tool to explore temporal behavior of envisioned distributed work architectures. We demonstrate a three-phase process that uses computational modeling and simulation of envisioned operations involving autonomous capabilities to evaluate the dynamic and emergent properties of envisioned operations. We discuss how computational modeling and simulation techniques can be used as a discovery tool to understand the impact of design decisions surrounding envisioned operations, such as the development of procedures for responding to and coordinating during foreseeable disruptions. This paper describes the application of this approach to address a current real-world design problem in the aviation domain, specifically the development of procedures and technologies for responding to communication failure in envisioned low-altitude short-range operations.
Background: Technology, Envisioned Operations, and Robustness
Cognitive systems challenges are fundamentally linked to time (Hollnagel, 2002). For example, expert performance involves taking action in such a way that workload is sufficiently spread through time. This involves looking ahead to anticipate what actions will need to be taken in the future (e.g., based on mental models of system dynamics) to help maintain performance despite potential obstacles (Hollnagel & Woods, 2005; Patterson et al., 1999). Field studies of cognition in risky worlds have shown that cognitive work is challenged by temporal demands, such as convergence of demands in time that can lead to workload saturation (Hollnagel & Woods, 2005; Pennathur et al., 2009; Xiao et al., 2004). The way in which demands escalate over time depends on the nature of the contingency as well as on the response of the practitioners, and how the response affects how the contingency progresses (Woods & Patterson, 2000). Field studies of responses to unexpected events (e.g., Woods & Patterson, 2000) have described the “escalation principle,” which denotes how, “as problems cascade, they produce an escalation of cognitive and coordinative demands that bring out the penalties of poor support for work.” Dynamics associated with escalation are in the form of cascades in the effects on the supervised system, which increases the demands for cognitive activity (more to monitor, more probing actions may need to be taken, as well as actions to protect integrity of the controlled process).
Keeping pace in distributed systems requires not only keeping one’s activities synchronized with the tempo of operations, but also maintaining synchronicity with the activities of other agents (Klein et al., 2005). As disruptions require escalations in cognitive activity, they also increase the demands for coordination (Woods & Patterson, 2000). For example, a flight crew’s work demands are determined by the need to keep pace with the dynamics of their aircraft and any surrounding traffic as well as by the pace at which automated systems reconfigure themselves (such as mode changes). Challenges can arise during anomaly response when distributed work requires synchronizing multiple interdependent threads of activity with limited time (Chow et al., 2000; Sarter et al., 1997; Woods & Hollnagel, 2006). For example, distributed work can break down when agents act upon stale information (Woods, 2020; Woods & Alderson, 2021). To stay synchronized in coordinated work, a common cadence that is fast enough to keep up with other agents and system dynamics needs to be reached.
Thus, in the context of work dynamics, feasibility and robustness of a system design can be evaluated as the ability of the system to maintain synchrony between the tempo of operations and the pace of the work system—where the work system is able to keep up with any escalating demands stemming from the work environment. Such synchrony (or asynchrony) emerges from the interplay between different threads of activity and the work environment. The timing and way in which actors respond to disruptions can have major downstream effects on the ability to stay in control, with accurate responses containing any cascading effects.
Evaluating Work Dynamics of Envisioned Operations
Designing new distributed work systems involves defining an architecture that includes descriptions of human and automated roles, procedures, and mechanisms for coordinating. Evaluating the work dynamics of future operational concepts is necessary to assure that a system design is feasible and robust to operate in various contexts. However, evaluating work dynamics for envisioned operational concepts is difficult for two reasons: (1) as previously discussed, understanding the dynamics requires capturing how interactions between system elements create emergent effects, and (2) the system does not yet exist and its design is inherently underspecified (meaning the descriptions of future operations lack detail), which leaves room for multiple interpretations of what the operations will look like.
The first challenge is the interplay between dynamic processes in the work environment and the elements of a sociotechnical system, which create emergent effects that are difficult to predict. The roles, responsibilities, and structural distribution of work that are defined as part of the system design influence how interactions play out over time and how escalations manifest as emergent effects. The interactions span both physical dimensions (e.g., the physics of the processes that need to be controlled, such as aircraft dynamics) and social dimensions (e.g., the intentions and actions of agents). Evaluating dynamics of work, therefore, requires holistic representations that capture the interaction between physical and social system elements (Lind, 2024).
The second challenge in evaluating work dynamics of envisioned operations arises from the inherent underspecification of the design and the associated inability to predict the range of behaviors that will emerge when the design is introduced into the work environment. There are several properties of the envisioned world problem (Woods & Dekker, 2000): (1) There are multiple versions of how the technology change may transform the operations (plurality), (2) each of the versions is vague about what it means to function in future operations (underspecificity), (3) the versions can be disconnected or contradict the research base (ungroundedness), (4) one cannot really know what one does not know (calibration).
Several studies have introduced model-based approaches for envisioning operations alongside the development of new technologies that will support those operations (Miller & Feigh, 2019; Rasmussen et al., 1994; Woods & Dekker, 2000). To facilitate the modeling of envisioned operations, cognitive systems engineering has advocated for describing systems in terms of the functions of their elements (including human and machine roles) and how they interact with each other (Woods, 2003; Woods & Hollnagel, 2006; Woods & Roth, 1994). This involves characterizing the purpose or reason for each system element rather than only modeling its physical components (Lind, 2024). To effectively integrate people, technology, and work, modeling in terms of what each element provides and affords relative to the goals of the system can reveal where there is functional coupling between system elements, often described as interdependencies (Johnson et al., 2014).
Several modeling frameworks have been developed for analyzing distributed work systems (Hollnagel, 2012; Leveson, 2001; Pritchett et al., 2014). The Functional Resonance Analysis Method (FRAM) (Hollnagel, 2012) identifies system functions and model dependencies and is used to analyze how variability in the performance of functions can create emergent behaviors (resonance). It is used extensively in the resilience engineering community (Patriarca et al., 2020). Likewise, Systems-Theoretic Process Analysis (STPA) analyzes system safety as a control problem and system operations as control loops (Ishimatsu et al., 2010). STPA uses models of the system to perform hazard analysis starting early in the design cycle of a new system (Leveson, 2011). Johnson et al. (2014) developed co-active design as a method for identifying and modeling interdependencies between roles.
FRAM and STPA have been used to conduct computational simulations of potential system behavior. However, none of the above methods explicitly evaluate how the cognitive functions of human and machine roles in the system dynamically interact with physical processes in the work environment. Work Models that Compute (WMC) is used to explicitly capture how work is “situated” in a dynamic environment (Pritchett et al., 2014). To do this, WMC explicitly models the interaction between environmental dynamics and the dynamics of the distributed work systems in terms of the actions performed by agents and their timing. WMC automatically captures the interactions between any of the components in the system without the need for explicit encoding of the interactions. Because WMC models functions and actions in computational form, WMC lends itself well to identifying emergent effects (such as patterns of activity) that arise from dynamic interactions between (potentially cascading) dynamics of the supervised or controlled system and the multiple threads of activity in envisioned distributed work systems (IJtsma et al., 2019).
In summary, the importance of pace and tempo in dynamic work environments creates a need for methods that can support system designers in anticipating the effects of technology on work dynamics. In this paper, WMC is used to simulate and explore the work dynamics for envisioned operations. We show how insight from the simulation of work dynamics can be used to assure envisioned operations are feasible and robust against known disruptions. While WMC has been used to simulate the performance of different work allocations (IJtsma et al., 2019; Pritchett et al., 2014), it has not been used to analyze the feasibility and robustness of an envisioned system in this way. In a broader sense, beyond the specific WMC framework, the paper introduces a three-phase approach for using computational modeling techniques to analyze work dynamics early in the design cycle of envisioned operations.
Computational Simulation of Work Dynamics
A dynamic work model is a representation of the interactions between work functions and the physical processes in the work environment, modeled in such a way that it can be used to evaluate emergent behavior. Work in dynamic environments like aviation is partially driven by environmental dynamics (as laws of physics). Static representations of work make it difficult to predict how: • Work and environmental dynamics interact. • Activity threads of different agents interact. • Behavior emerges from such interactions.
To evaluate work dynamics and the ability to keep pace with escalating work demands, a distributed work system can be modeled as multiple dynamical processes: the dynamics of the work environment (e.g., aircraft dynamics) and the dynamics of the work system (e.g., the pace by which various work processes are performed, such as anomalies being diagnosed and interventions being formulated and implemented). By simulating the interaction between the processes, emergent behavior can be captured due to the modeling of: 1. An environment-driven effect where the events in the work environment partly determine the required pace of activity for practitioners. Dynamics of the work is influenced by the need to respond to events in the work environment, where these events dynamically evolve over time. For example, the dynamics of two aircraft in conflict (meaning they are projected to fly dangerously close to each other) determines what and when actions need to be taken to prevent a loss of separation. 2. An action-driven effect where the actions of practitioners partly determine how the contingency progresses. Here, what and when actions are performed will alter dynamics in the environment, as actions act upon the work environment. For example, early detection of a conflict and consequent resolution action will result in a different aircraft trajectory than a late detection and response, and, with it, different downstream events.
The cross-interaction creates emergent effects: dynamics of the work environment are influenced by responses of the work system as agents act upon the work environment, and vice versa.
A three-phase approach is described in the following paragraphs to develop such a dynamic work model, which is illustrated with a case study in the remainder of the article. The three phases of the approach are shown in Figure 1. Numbers in the figure correspond to the descriptions below, and elements of the figure will be elaborated upon in subsequent sections. 1. A conversion from paper-based representations and verbal descriptions of envisioned systems to a computational form that is suitable for fast-time simulation (i.e., an encoding process to build the computational model). 2. The development of scenarios and testing conditions to simulate. 3. The analysis of emergent behaviors through simulation and the use of the model to make inferences about the performance of the distributed work system (i.e., a decoding process to interpret the results relative to the envisioned operations). Evaluation Approach using Computational Modeling and Simulation.
These three phases are not to be interpreted as a linear process—an analysis with modeling and simulation techniques typically involves many iterations, going from initial high-level representations of system interactions and progressively becoming more detailed and specific with each iteration. In this method, the three phases help frame an analysis of the feasibility and robustness of envisioned systems in light of their work dynamics.
Phase 1: Converting Paper-Based to Computational Representations
The first phase in modeling and simulation of envisioned operations is to identify the interactions between work functions and the processes in the work environment and model the work functions and processes in a way that they can be simulated to evaluate emergent behavior. Functional modeling can support the envisioning process and help anticipate what transformations may occur as a consequence of technological change. For example, descriptions of functions and their dependencies can support: • Identification of new functions that may arise or need to be supported as a consequence of technological change. • How those new functions emerge into new roles. • How dependencies between functions can create new complexity and new pathways to failure.
What functions need to be modeled depends on the specific purpose of the evaluation. In any case, representations of the system’s functions should have a clear mapping to the envisioned operations. A review of documents that describe the system’s operations, goals, and/or components is an important source. Concepts of operations usually describe the system and its functions, which can provide a basis for computationally modeling those functions. Concepts of operations are not necessarily the only source of information in this phase; data for the computational model can come from paper-based cognitive engineering techniques like the critical decision method (Klein et al., 1989). A computational model of work in WMC formalizes each function as one or more transformations between input and output. The input and output are in the form of environmental parameters, where the function represents a transformation of the environment’s state.
Functions are generalizable descriptions of possible change, whereas actions describe the exact change and details of attaining it (Lind, 2024). For example, UAM operations aimed at achieving safe and efficient flight will need flight control functions that can be achieved by taking off, managing route progress, changing heading, landing, etc. Control of flight is generalizable and can be achieved by performing any of the actions described above. The amount of detail in the actions directly relates to the modeling objectives. If the purpose is to understand the system dynamics during a takeoff or landing, then a higher level of fidelity is required for modeling takeoff and landing. If the modeling objective is to see how the aircraft performs while in flight, then takeoff and landing can be relatively lower fidelity.
WMC uses three main computational structures: resources, actions (as specific instantiations of functions), and agents. While a brief overview of the computational structures is provided here, a more detailed overview can be found in earlier publications on WMC (Feigh et al., 2014; Pritchett et al., 2011, 2014).
Actions represent the work performed by the agents in the system. Work is purposefully modeled independently from agents to allow for efficient testing of different work distributions. An action represents the actualization of a function in a more specific context. Actions are modeled computationally in two parts. One set of transformations represents the work that an action does on the work environment; that is, how it takes input from the work environment—for example, an uncrewed aircraft's UA’s position and airspeed—and transforms it into output—for example, the UA’s expected time of arrival. The second set of transformations represents how the action determines its own timing, where the timing is determined either by other actions (it is activated based on another, prior action being taken) or in response to events in the environment (e.g., the landing action is scheduled more frequently as the UA approaches the vertiport).
Resources represent the information necessary to perform work, both as information that the activity responds to and acts upon. Resources in WMC are elemental variables that represent individual states of the environment. For example, if an environment has an aircraft, resources capture the altitude, airspeed, and attitude.
Finally, agent models in WMC are simple relative to other multi-agent simulation frameworks. They are considered as actors that get passed actions to perform, with the bulk of the work model already captured in the action and resource structures. While agent performance can be modeled, WMC is also typically run with an assumption of “perfect” agents—that all agents start performing the work immediately when called upon, and without errors (contrast this with an agent model that simulates internal constraints regarding the number of actions that can be executed simultaneously). This means that emergent effects identified from simulations are first and foremost representative of how actions and environment interact, and how demands arise and can escalate based on what and when activities are performed, without making modeling assumptions about agent performance.
The output of this phase is a semi-completed dynamic work model that is a partially underspecified representation of the envisioned operations, as any envisioned concept inherently has incomplete or conflicting information. Some underspecificity can be addressed through more directed literature surveys. Discussions with subject matter experts, if available, are another source of information that can help address gaps in paper-based descriptions. Additionally, remaining underspecificity can be addressed by documenting assumptions or by creating different possible alternatives for simulation, as discussed in the next phase.
Phase 2: Developing Scenario and Testing Conditions
The next step is to define one or more scenarios that are characteristic of the complexity and challenges of real-world operations yet keep simulation efforts manageable. We have found that an ideal scenario for evaluating work dynamics contains some aspect of temporal challenge, such as a situation where actors need to act quickly to keep up with environmental dynamics. In addition, an ideal scenario contains some degree of uncertainty that the actors are expected to handle. Lastly, if there is potential for cascading effects, these should be considered in the scenario selection so that they can be evaluated. Cascading effects can also emerge from the normal scenario design without being explicitly added.
Any remaining underspecificity in the model relative to what is required for simulating a scenario can be addressed by making and documenting assumptions. When there are multiple versions of envisioned operations (i.e., the plurality aspect of envisioned world problems), the differences between these versions can be the basis for testing conditions, as input to multiple simulation runs, where computational simulation can be used to compare alternate versions. Variables can be assigned different possible values, representing different assumptions about the envisioned operations, often in the form of different system architectures.
Phase 3: Simulating Work Dynamics and Analyzing Emergent Behaviors
Simulation output provides insight into how interactions play out dynamically and how they create desirable and undesirable emergent effects. The results from the simulation can be related to and interpreted relative to the existing research-based patterns from cognitive systems engineering (Woods & Hollnagel, 2006) to compare the emergent properties and take steps to mitigate any adverse effects. In addition, results can be fed back to the prior two steps to further develop and specify the model and the operations (see Figure 1).
The tempo of operations is operationalized in the simulation as the number of actions within a given time window. Increasing the number of actions to be performed within this window represents an increase in the temporal demands for the actor: they need to perform more actions in the same amount of time. The tempo of operations can also increase when the time window decreases: the number of actions does not increase, but the time available to perform them decreases.
Phase 3 can begin even if phases 1 and 2 are not complete—in fact, we find that early simulations help reveal where further specificity is needed, which helps direct further document review and stakeholder engagement to clarify missing elements in the concept of operations and identifies additional elements to account for in our model. Simulating models from the earliest stages of envisioning supports iteratively designing and evaluating the concept of operations. One can then progressively extend the models as functions, architectures, and procedures are further specified.
During this phase, several higher-level questions should be asked regarding how the simulation results can help guide the design process. Example questions: • Are there any assumptions that need to be concretely specified for the system? • Are there instances of undesired emergent properties such as temporal convergence of task demands (indicative of a potential for workload spikes)? • If yes, what are pathways and trade-offs associated with mitigating undesirable effects?
Simulation results can be interpreted through three different lenses: evaluating the feasibility of work dynamics of envisioned operations, comparing alternate versions of envisioned operations, and enhancing the robustness of envisioned operations. The three lenses represent different ways of interpreting the output from simulation. In most cases, a combination of these lenses can be used to provide a comprehensive analysis of an envisioned work system
The first lens interprets the results with the aim of evaluating envisioned operations for feasibility. In other words, it conducts simulation to understand to what degree the envisioned operations, as described on paper, actually achieve the desired outcomes. This lens highlights how, when moving from static to dynamic representations of envisioned operations, unforeseen effects may arise that warrant adding further specificity or other modifications to the concept of operations. Compared to paper-based and qualitative descriptions of work, developing a computational work model requires making work functions explicit. While existing qualitative descriptions are a valuable input, it is often found that these descriptions are underspecified relative to what is needed to make a concept of operations “run” dynamically. Hence, through constructing and simulating the computational model, this layer involves systematically making and logging extensions to the existing descriptions to make the concept of operations feasible in a dynamic sense. This is an iterative process in which work is modeled, initial simulations are run, and output is analyzed to see whether desired outcomes are achieved.
The second lens interprets results with the aim of comparing alternate versions of envisioned operations. Comparing the results of multiple simulations can provide insight into system behavior that can be used to facilitate discussion among stakeholders. The versions to simulate are typically grounded in discussions with stakeholders.
The third lens interprets results with the aim of understanding sources and risks for enhancing robustness of envisioned operations. It is associated with exploring where work dynamics may break down; that is, where can we find vulnerabilities in the design? In terms of work dynamics, this can include identifying potential for asynchrony, such as the activities of agents being out of sync with each other or failing to keep up with the escalating demands of a contingency. This lens can help reveal pathways for increasing the system’s capacity to adapt to challenging situations, for example, by modifying how work is distributed to allow for faster responses, or identifying new functions to better synchronize multiple threads of activity.
Case Study in Urban Air Mobility
To demonstrate the approach, we discuss a study performed by the authors that applied the approach to analyze envisioned short-range low-altitude air transportation concepts, such as UAM. These concepts of operations are being developed by gathering input from different stakeholders in the industry (FAA, 2023). One critical function for enabling these concepts is contingency management. An implicit assumption in several of the operational concept descriptions is that autonomous capabilities will manage disruptions in a decentralized manner using predefined procedures. However, humans play a critical role in managing anomalies in current aviation operations, and further development of these concepts must consider how human roles can coordinate while keeping pace with potentially escalating demands.
Our analysis focused on a lost command and control (C2) link contingency, which has been studied in the context of uncrewed aircraft systems (UAS), but has received minimum attention (Boeing and Wisk, 2022, RTCA SC-228, 2023; Gregory et al., 2021; Woods & Balkin, 2018). Several options can be identified for the lost C2 link (LC2L) aircraft’s response, drawing from sources such as RTCA SC-228 (2023) guidance, FAA procedures for lost two-way communications in piloted aircraft, and discussions with stakeholders. The aircraft can continue to fly along its predefined and approved flight plan or can divert to an alternate vertiport, for example, the nearest (but could also include other onboard logic for dynamically choosing what to do). A reason for continuing to fly along its predefined route is that this flight plan is (or should be) strategically deconflicted. If no further anomalies occur and all other airspace users conform to their approved flight plans, the aircraft should not lose separation with other aircraft. This envisioned response is similar to FAA guidelines for two-way radio communications failure in the current air traffic system, which note that under Instrument Flight Rules (IFR) an aircraft should continue along the last cleared route (U.S. Department of Transportation, FAA, 2023). However, in a remotely piloted system, this creates a higher level of risk, especially when C2 link loss occurs early on a long flight and the aircraft needs to fly without a link for an extended period of time.
As an alternative version of envisioned operations, FAA Job Order (JO) 7110.724 (2016) recommends flight termination if a positive link cannot be established. This order provides more information on the exact procedures to try and acquire clearances for an aircraft with a pilot in command. Still, it has no guidance about procedures if the aircraft is remotely piloted. Similarly, RTCA SC-228 (2023) also recommends diverting the aircraft. The second option of diverting could reduce the duration that an LC2L aircraft is in the airspace and reduce the likelihood of further anomalies complicating the response. However, diversion to alternate vertiports can create conflicts with other traffic that pose a challenging coordination problem. Procedures for surrounding traffic for coordinated responses to potential conflicts with an LC2L aircraft have remained relatively underspecified.
Surrounding traffic also has several options: it can continue to fly along the approved flight plans while being extra vigilant for potential conflicts. Should conflicts arise, aircraft are envisioned to be equipped with detect and avoid capabilities, and those should be able to handle tactical conflict management (Boeing and Wisk, 2022). Alternatively, to reduce risk, surrounding traffic can perform tactical replanning to divert other traffic away from the LC2L aircraft. This could create a larger buffer around the contingent aircraft and reduce reliance on detect and avoid capabilities as a last line of defense. It does require a more concerted effort from impacted pilots and fleet operators to reroute surrounding traffic.
In the following sections, we illustrate how we applied the three-phase approach to evaluate the work dynamics and derive insight for the further development of this concept of operation.
Phase 1: Converting Paper-Based to Computational Representations
The aim of the case study was to evaluate envisioned procedures for a LC2L aircraft and surrounding traffic to coordinate and synchronize their responses. To develop a computational model, the analysis began with a document review and stakeholder engagement. Document review started with the written descriptions of the envisioned system, particularly the concept of operations like the NASA UAM Concept of Operations (ConOps) (NASA, 2020), FAA UAM ConOps (FAA, 2023), and the Boeing and Wisk Uncrewed UAM ConOps (Boeing and Wisk, 2022). Underspecificity in these documents (relative to what was needed for the work model) were addressed through more targeted reviews of related documents (e.g., RTCA SC-228, 2023; Woods & Balkin, 2018) and through conversations with subject matter experts, including several engineers directly involved in the development of UAM operations.
From the document review and stakeholder engagement, two computational work models were created: one for aircraft operations and one for airspace management. Key dynamic processes in the work environment are the dynamics of the aircraft in the affected airspace. Aircraft in the scenario must keep moving due to physics of flight and, therefore, cannot stop while humans execute procedures and complete checklists. By modeling how work actions interact with these dynamics, emergent behavior during contingency management can be made explicit. The aircraft dynamics were modeled using linear equations of motion, approximating each aircraft as a point mass. Forces and acceleration were calculated based on inputs from the control systems, and resulting airspeed and position were calculated as the simulation time progressed. Several control loops were implemented to maintain a target airspeed, altitude, and heading. In addition, actions were modeled that would simulate how an aircraft flies through a set of target waypoints (as specified in a flight plan). While linearity and point mass make this simplified aircraft model sufficient to answer the research questions, more detailed flight dynamics can be easily implemented in the simulation framework if required.
List of Actions Modeled as Part of the Aircraft Operations Work Model in WMC Simulation Framework.
List of Actions Modeled as Part of the Airspace Management Work Model in the WMC Simulation Framework.
To complete the models, the actions need a duration, the time taken to perform the action, and the roles of agents performing the action. The durations are approximated using the information gathered from stakeholder discussions and subject matter interviews. The durations are based on the best guesses of subject matter experts based on current-day operations and should be subject to revision as there is more information about the envisioned operations. A subset of system actors involved in responding to a loss of C2 link contingency was identified from the source ConOps documents: the aircraft and their onboard autonomous capabilities, pilots of crewed aircraft (pilot-in-command, or PIC), remote pilots for uncrewed aircraft (remote-pilot-in-command, or RPIC), fleet operators (FO), providers of services to UAM (PSU), vertiport operators, and vertiport automation. Agents outlined in Table 1 and Table 2 are ascribed authority to perform these actions as a default role description. These role descriptions are based on available information in the concept of operations and discussion between the authors and engineers involved in the development of the UAM operational concepts. Because WMC is set up with separate agent and action models, the roles of agents and the actions they perform can easily be changed.
The actions that are primarily performed by aircraft automation need human supervision, and therefore, an additional monitoring action is created automatically. It was deemed that, relative to other, higher-level actions, the waypoint calculation and heading change actions require lesser degrees of human monitoring and, therefore, do not have additional monitoring actions. The contingency plan execution action for the contingent aircraft cannot have human supervision due to the loss of link and is therefore also excluded from monitoring.
Phase 2: Developing Scenario and Testing Conditions
The present study modeled a scenario with five aircraft: two remotely piloted using the C2 link and three with an onboard pilot. Figure 2 shows the original filed flight plan for all five aircrafts at the start of the scenario based on the envisioned DFW airspace by NASA (Verma et al., 2022). The figure shows that the flight trajectories use the same routes but are strategically deconflicted. Aircraft with the call sign “Helio” takes off from the Frisco vertiport (FSC) and flies through six waypoints before reaching the Dallas vertiport (4DT). Aircraft “Vista” travels from the Dallas vertiport to the T57 Garland vertiport. Aircraft “Black” travels in the opposite direction of “Vista” from T57 Garland vertiport to Dallas vertiport. Aircraft “Venus” and “Hyper” are traveling to the Frisco vertiport from T57 Garland vertiport and Dallas vertiport, respectively. At the start of the scenario, Helio and Vista are midflight, and Hyper takes off from the Dallas Vertiport. Black and Venus start flight at about 170 and 350 seconds into the scenario, respectively. Midflight, the remotely piloted Helio loses its C2 link with the remote pilot in command (RPIC Helio). Helio has a pre-determined contingency plan loaded for a loss of a C2 link, as described in one Uncrewed UAM ConOps (Boeing and Wisk, 2022), which is automatically activated by the aircraft. This alternate plan is to either continue along its original flight plan or landing at one of four pre-coordinated vertiports, selected based on the aircraft position. Flight paths during Baseline condition.
While the aircraft executes the contingency plan, the system reconfiguration procedure is envisioned as the RPIC alerting the PSU. The PSU network then broadcasts the lost C2 link alert and the aircraft’s expected trajectory to impacted actors (PSU(s), vertiport(s), fleet operator(s)). The affected traffic alters its trajectories as needed to avoid conflicts with the contingency flight trajectory of the LC2L aircraft. This replanning involves coordination and negotiation between PSU(s), vertiport(s), fleet operator(s), and pilot(s) to finalize new operational plans.
Assumptions Made to Create a “Runnable” Model.
While some of the underspecificity was directly addressed by these assumptions, in other cases, we found the underspecificity to reveal more fundamental questions surrounding the architecture and procedures for responding to lost C2 link contingencies. This underspecificity was a starting point for modeling more than one version of envisioned operations. These versions, in this case study, evolved around two key (currently unanswered) questions: • What should be the aircraft’s lost C2 link contingency plan? • How should the other actors in the system respond?
Alternate System Responses to Lost C2 Link Contingency. A Baseline Condition was Also Simulated With No Contingency (i.e., Nominal Operations).
Phase 3: Simulating Work Dynamics and Analyzing Emergent Behaviors
The four system architectures summarized in Table 4 were simulated, and the output from the simulation was analyzed to gain insights into the performance of each envisioned system response. For an interactive version of the results with the ability to view how the flights move with time or to look at other results like speed changes, please visit https://huggingface.co/spaces/CSEL/aam-demo. In successfully iterating and simulating the computational model from start to finish, the assumptions mentioned in Table 3 were noted to be crucial and needed to be concretized for initial system feasibility.
First lens: Identifying underspecificity and creating feasible work dynamics
Figure 2 shows the trajectories for the baseline condition with no loss of the C2 link. Figure 3 shows the flight trajectories for the condition in which the LC2L aircraft diverts to the alternate vertiport and the surrounding traffic responds (Divert & Respond in Table 4). The results show how the aircraft-level response of diverting (in this case, Helio as the LC2L aircraft) to the nearest vertiport creates a need for a system-level response that involves the aircraft Vista rerouting to avoid a conflict or Hyper slowing down to avoid a conflict. This illustrates the significant potential for cascading effects. In this case, an LC2L aircraft requires other aircraft to amend their routes, which can create secondary conflicts that need to be resolved or changes in airspace demands that must be accommodated. If one aircraft alters its operational intent, system-wide replanning may be necessary to maintain safety. Note that this may apply in more than just contingency scenarios. Final flight paths during Div&Resp conditions.
The reroute was envisioned to involve the aircraft Vista executing a holding pattern to allow the contingency aircraft to pass in front of it, yet involved coordination between the pilots of Vista, the fleet operator, PSU, and the vertiport operator to obtain a new operational intent for Vista. However, we found this static representation to be too simple relative to the dynamics at play: when a new OI is proposed by Vista, the aircraft is located between the NTHW and NTI waypoints, where it planned to hover. However, by the time the plan was accepted, Vista had moved along its original path. This created a possible loss of separation at the landing vertiport and the need for Vista to execute a go-around. This highlights the adaptive trap of working at cross purposes; the local adaptations of the high automation authority to the loss of the C2 link are counterproductive to the overall system safety.
Even with fast response times, the lost C2 link procedures envisioned in this study have a period of time in which secondary aircraft continue to fly on their original (approved) operational intents. In contrast, the LC2L aircraft (automatically) implements its preprogrammed contingency response 20 seconds after noticing the loss of link. Thus, when the LC2L aircraft automatically implements an alternate flight path, there is a risk of loss of separation while secondary aircraft are reconfiguring their own flight paths to accommodate the LC2L aircraft. Thus, while generating new operational intents for airborne aircraft during a contingency, replanning must account for the time it will take to plan and negotiate reroutes (Fernandes et al., 2018; RTCA SC-228, 2023). During this time, aircraft continue to fly according to their currently confirmed operational intents. Any reroutes need to account for the flight progress on this original path before the aircraft will receive its amended operational intent. Thus, the simulation revealed the need for every operational intent amendment to account for a Minimum Trajectory Negotiation Duration (MTND) (Fernandes et al., 2018) or Lost C2 Link Decision Time (RTCA SC-228, 2023), which is yet to be defined for short-range low-altitude operations like air transportation concepts like UAM concepts.
Second lens: Comparing Different Versions of Response
Four versions of the envisioned operations were simulated to explore alternate responses to a lost C2 link contingency, as shown in Table 4. Figures 4, 6, and 7 show action traces for the Baseline, Divert & Respond, and Continue & See cases, respectively. Each action trace shows the number and kind of tasks performed by each agent in the simulation at each point in time. Figure 5 shows the different actions in the simulation and the color pattern used to represent them in the action trace. Note that the axis for each agent is different due to the vastly different number of tasks executed by each agent in the simulation. The orange line in each action trace indicates a moving average of the overall taskload for the agent in the previous 60 seconds. This line can reveal instances of high taskload, such as the one in the PSU or PIC Vista action trace in Figure 6. A small section of the RPIC Helio action trace is enlarged to represent the orange average line along with instances of high and low taskload. In this section, it can be seen that there is an instance of low taskload represented by the green highlighting of the orange line and a high taskload represented by the red highlighting of the orange line. The increase in taskload can be attributed to the added actions performed just before the red highlighting starts, each vertical bar is one action. The Divert & Respond condition (Figure 6) shows a higher taskload for multiple agents than the baseline condition and reveals at least seven taskload peaks, indicating a rise in taskload when there is a diversion due to unexpected events. This indicates a possible tradeoff between the taskload associated with responding to a diverting aircraft versus uncertainty. The alternative would be to allow the aircraft to fly as per the original operations plan as on the Continue & See condition (Figure 7) but then risk the possibility of an uncontrollable aircraft in the airspace for a longer duration. Action traces with moving averages in the Baseline condition. Action traces with moving averages and taskload metrics in the Divert & Respond condition. Action traces with moving averages and taskload metrics in the Continue & See condition.
Figure 8 shows a comparison of the number of taskload peaks by condition and agent. A taskload peak was defined as an instance of a higher taskload in the moving average for the agent than the taskload measure around that instance. As expected, the taskload for the Continue & Respond and Divert & Respond conditions have the highest number of peaks due to the number of tasks required to select and implement the system response. The RPIC Hyper, PSU, and Vertiport agents have an additional spike in taskload for both of the Response conditions. The PIC Vista agent has an additional taskload peak for the Divert & Respond condition, whereas the PIC Black agent has an additional taskload peak for the Continue & Respond condition due to the need for the Vista or Black to respond and increase separation. The RPIC Helio agent has the same number of taskload peaks in all four conditions. The overall peak taskload value is higher in the Respond conditions than in the others due to the increase in the number of tasks required to increase separation in the airspace. Number of Taskload Peaks by Condition for Agents.
Simulation of the four alternate responses revealed several factors to consider in further development of each alternate. There is a fundamental tradeoff between local responses (in this case, having the LC2L aircraft continue to fly, creating potential for significant uncertainty about the aircraft’s state and intentions) and system-level response in terms of the task load for all parties involved. Local response of the aircraft affect the amount of time there is an uncontrollable vehicle in the airspace. System-level responses show a significant increase in the taskload peaks when there is a need for an increase in separation. For instance, in further development of the responses, lost link aircraft response to divert might need a lot of system-level replanning and has a potential for cascading in high airspace traffic situations. Instead, in cases of high airspace traffic, continuing along the initially deconflicted path may be more efficient in minimizing system-level complexity.
Third lens: Varying the timing of the contingency
For the third computational experiment, the timing of the contingency was changed. In particular, we were interested in understanding the system response when the contingency occurs when the aircraft is close to a decision waypoint, including what coordination demands may arise in a time-critical event. At 420 seconds, the aircraft was identified to be close to a decision waypoint (NTI noted in Figure 10), which determines the diversion vertiport for the aircraft. The contingency time is therefore moved to 420 seconds to see how this extreme change affects the system’s contingency response. A bird’s eye view of the aircraft trajectories for a loss of C2 link occurring close to a decision point is shown in Figure 10. Figure 9 shows the difference in the time available between a loss of C2 link and Helio descent in the normal case (a) and when it happens close to a decision point (b). The window to communicate the loss of link before Helio starts the final sequence is reduced heavily (red blocks in Figure 9), leading to the system falling behind the LC2L aircraft. System state and tempo of operations when (a) loss of C2 link happens close to a decision point, (b) loss of C2 link happens farther away from a decision point.
By varying the time of the C2 link event, the simulation helped uncover a potential for a coordination surprise (Patterson et al., 1998) as information about the loss of link event is not shared in a sufficiently timely manner. Figure 10 shows how Vista has to do a go-around as two aircraft were landing simultaneously due to a coordination surprise. In this situation, the time taken to confirm and communicate the loss of link caused the vertiport to find the aircraft in its airspace before receiving information about the loss of link having occurred and the aircraft’s planned diversion. The LC2L aircraft’s decision to divert and location relative to the diversion waypoint when the contingency occurs leads to a sudden turn followed by an initiation of the landing sequence. To the vertiport, this is a surprise because the pilot is still confirming the contingency and has not yet communicated the contingency status to the rest of the system. To add to the complexity of the scenario, there is already an aircraft on approach to the same vertiport. The result is a loss of separation, which increases the risk of midair collisions. This illustrates how the inability to keeping pace in coordination (not sharing relevant information in a sufficiently timely manner) can result in a further increase in task demand when the vertiport must then act quickly to redirect an aircraft and avoid a collision. Bird’s eye view of contingency at t = 420.
This illustrates how important temporal factors are, by showing the relation between slow versus fast responses. The aircraft automation is fast to confirm and react to the contingency, while the rest of the system is still confirming the loss of link. The pace of automation reaction combined with the aircraft continuing to move towards its destination causes the rest of the system to fall behind the tempo of operations. It also highlights the coupling of the different actions in the system and how they should be in sync or allow other actions to catch up or slow down. The trade-off associated with the speed and implementation of the response is also important; if aircraft automation waits too long to act to allow the system to catch up, it might be too late before the aircraft takes action, leading to escalating demands. Similarly, if the response is too fast, it can cause unnecessary burdens to the system in the case of false alarms.
Discussion
Computational modeling and simulation using WMC can evaluate the distributed work system for feasibility and enhance the robustness of the system by capturing the dynamic interactions and emergent effects. The case study shows how the simulation of envisioned contingency management operations can help evaluate interactions and identify requirements, like minimum trajectory negotiation time and taskload peaks, that are not previously accounted for in the static representations of the design. The case study reveals how simulations can help quantitatively and explicitly show the feasibility of envisioned operations and the assumptions required to make the operations feasible.
The simulations revealed how the success of contingency management is determined by the ability of procedures to stay in sync with an escalating tempo of operation. The dynamics of the short-range low-altitude air transportation systems are fast, resulting in the system’s response to contingencies being highly time-pressured. Simulation of the aircraft dynamics relative to the envisioned procedures showed that system actors can quickly lose control of a contingency when the system’s response is slow relative to the tempo of operations. This is indicated by the coordination surprise event created due to delayed detection, confirmation, and communication relative to one aircraft’s diversion and approach to the vertiport.
The new application using our three-phase process to model and evaluate the lost C2 link scenario using the WMC simulation capabilities helped address the complexity of a distributed system responding to a dynamic event, which led to documentation of assumptions, models of the tasks under investigation and feasible distributed work system designs, as well as some of our functional and performance requirements. Two specific recommendations for further development of the envisioned UAM operations were derived: 1. If system coordination is slow relative to the decision-making of the LC2L aircraft, it creates potentially unsafe situations. When reroutes were not approved or implemented quickly enough, separation was lost between the LC2L aircraft and a secondary aircraft. Because coordination demands can be substantial relative to the pace of events (or the available time) when a lost C2 link occurs, increasing system robustness should focus on ways to speed up coordination or slow down the aircraft OI change to allow the coordination to catch up to the aircraft’s decisions. 2. If actions of aircraft automation are not synced with the rest of the system, it can lead to a high-pressure situation where the humans have to make quick decisions to keep the system safe. We found that in certain conditions, coordination surprises can occur when the LC2L aircraft deviates from its prescribed trajectory before other agents are aware of the contingency. Thus, further development should consider how other actors can anticipate automation actions during contingencies, for example, by designing automation behavior to be predictable or creating mechanisms for other actors to efficiently monitor and diagnose automation behaviors.
Insights from the model are only useful when they accurately reflect the work functions for the operations. There are two types of validity here: one is with respect to what is envisioned by the designer, that is, the models should accurately reflect the architectures and procedures that are part of paper-based models or discussions with stakeholders. The second one is with respect to how work is actually done, which does not always correspond to work as envisioned (Dekker, 2006). While in conceptual design phases, work-as-envisioned is usually all there is (there is no work-as-done until the new operational paradigm is realized), there are some ways that work-as-envisioned can be made more accurate. Interviews and observations can be conducted with experts, possibly in analogous or legacy work domains, to gain insights about work demands (Miller & Feigh, 2019) as input to the modeling. In addition, patterns from the research base can be used to amend descriptions of work in documents or stakeholder interviews. For example, while the models presented as part of the case study were first and foremost built on documents and interviews that reflect the operations envisioned by the aviation community, additions can be made to the model based on patterns in cognitive systems engineering that better reflect true work demands. For example, WMC has earlier been used to model monitoring demands, based on first principles of cognitive systems engineering (authority-responsibility mismatches, see Pritchett et al., 2014) and communication demands (based on patterns of implicit and explicit communication, see Barrett et al., 2024). In future extensions of the WMC framework, similar patterns from naturalistic studies of distributed work can be used to more accurately reflect the real-world demands beyond a first-order envisioning based on documents and interviews.
Gaps between work-as-done and work-as-envisioned are often the result of adaptations in strategies to cope with local demands that were not foreseen as a part of the envisioning process (Woods & Dekker, 2000). While post-deployment surprise is arguably inevitable in any kind of envisioning for complex and dynamic work domains (i.e., no envisioning can be perfect), modeling and simulation can provide insight about the dynamics of demands that goes beyond what one could gain from other methods for envisioning. (e.g., methods used in Miller & Feigh, 2019). As such, it can be one tool in an arsenal of tools and techniques to support more accurate envisioning and system design. Even when a model reflects work-as-envisioned more so than work-as-done, many things can be learned from dynamic simulation of the work-as-envisioned, such as the potential risks and pitfalls around work dynamics that practitioners would need to cope with and adapt to (such as the two insights and recommendations discussed earlier in this section). This type of insight can bring the overall envisioning process (beyond the models) closer to recognizing the true work demands that will be imposed on the practitioners for a given system design and procedure. To make the tool more useful in an evolving envisioning process, future development of this approach can focus on reducing the effort and time required for revising and extending the models, so that new information or new ideas can be incorporated efficiently as—throughout the design process—more is learned about the work demands and what procedures or system designs could be useful.
Thus, this new application of WMC, as used in the paper, is meant as a discovery tool for envisioning new system designs and procedures and is complementary to (not replacement of) other cognitive systems engineering techniques or the involvement of subject-matter experts and practitioners in the design process. Computational simulation of work dynamics provides only perspective and also has limitations like any model. For instance, it does not capture how practitioners may adapt to work demands and is limited to scenarios that the modeler simulates. Hence, a comprehensive feasibility and robustness check of envisioned operations should also involve other techniques, expertise, or tools. For example, subject-matter experts can help determine what scenarios to simulate and what constitutes sufficient exploration of variations on the base scenarios to deem a simulation analysis complete. As mentioned before, an ability to efficiently reconfigure models would facilitate the exploration of a wider range of scenarios and system designs.
In conclusion, computational simulation of work can be used as a discovery tool to explore the dynamic of envisioned operations without the need to build expensive prototypes or wait in the design process until various system elements can be integrated. It provides a way to envision how advancements in technological capabilities (e.g., new autonomous capabilities) can be integrated into operations that involve distributed work. We argue that the development of new technological capabilities and the envisioning of new forms of operations should be done in conjunction, and computational modeling and simulation techniques are a way to explore dynamic behavior in the earliest stages of technology development. Simultaneous development helps operational concerns and constraints inform the design of the new technologies and, vice versa, helps operations consider early-on the implications of new technologies making their way into operations.
Footnotes
Acknowledgments
This work was performed under NASA contract number 80NSSC23CA121, with Jason Prince serving as NASA technical monitor. The authors greatly appreciate the inputs and feedback provided by the NASA team. The authors would also like to thank the Mosaic ATM team members and other WMC developers.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was performed under NASA contract number 80NSSC23CA121.
