Evaluation of Cognitive Skill Degradation in Flight Planning

Abstract

This work aimed to identify cognitive skills associated with flight planning, suggest which skills might be susceptible to skill degradation, and investigate the effects of cognitive skill degradation over time. Information automation systems offload cognitive tasks to reduce workload and error. However, the same phenomena seen with physical skill degradation in highly automated aircrafts may also occur when automating cognitive tasks. Two studies were conducted. An applied cognitive task analysis identified cognitive skills in flight planning. An empirical evaluation examined whether some of those skills were susceptible to cognitive skill degradation over time when using automation. Participants were placed into three groups. After conducting a flight planning task manually, groups differed in the next three practice trials: manual, alternating between manual and automation, or only with automation. Finally, all groups conducted the task manually again. Trials were separated by 2 weeks. The automation group showed the most performance degradation and highest workload, while the manual group showed the least performance degradation and least workload. Automation use did not provide the practice needed to mitigate cognitive skill degradation. Analysis of the impacts of information automation on cognitive performance is a first step in understanding the root causes of errors and developing mitigations.

Keywords

cognitive processes applied cognitive task analysis aviation information automation

Introduction

The objective of this work was to identify the cognitive skills associated with flight planning, which skills might be susceptible to skill degradation, and the effects of cognitive skill degradation over time. The goal of the research was to determine if automation has an effect on cognitive skill retention and performance. In order to do this, one must understand what cognitive skills are involved in flight planning and how those skills are affected by the use of automation.

Reliance on increasingly advanced automation, increases in long-haul flights, and airline policies have significantly decreased the opportunities to fly an aircraft manually. This affects a pilot’s chance to practice and maintain manual flying skills (BASI, 1998; Casner et al., 2014; Gillen, 2008; Mengelkoch et al., 1971). This lack of practice may lead to decreased situation awareness (SA), overreliance on automation, and the potential for physical and cognitive skill degradation (Casner et al., 2014; Dudley et al., 2014; Parasuraman & Riley, 1997; Parasuraman et al., 2000). Pilot physical skill degradation has been studied extensively, where the loss of manual flying skills due to infrequent practice may diminish the crew’s ability to take over when automation fails or is turned off (Billings, 1991; Ebbatson, 2009; Prophet, 1976; Tenney et al., 1998). In aviation, there is a concern that the same phenomena seen with physical skill degradation may be relevant for cognitive skills (Casner et al., 2014).

Cognitive skills are mental resources used in the application of knowledge to a task utilizing basic skills such as thinking, reading, learning, remembering, and reasoning (Lourda & Jose, 2019; Singley & Anderson, 1989). Tasks and cognitive processes that are affected include: memory, recall, calculations, situation assessment, decision-making, alert comprehension, future state prediction, and action plan generation. Cognitive skills require declarative and procedural knowledge.

Cognitive skill degradation is the reduction of thinking, reasoning, and decision-making skills. Without practice, cognitive skills are susceptible to degradation—fully automating a function eventually could lead to skill decay due to forgetting and lack of practice (Argote et al., 1990; Ebbinghaus, 2013; Rose, 1989; Wickens, 1992). A study by Argote et al. (1990) indicated a rapid rate of learning depreciation over periods of nonpractice, in some cases as much as 97% following a 1-year period. Furthermore, cognitive skill is linked with both workload and level of expertise.

There can be an interaction between cognitive skill and mental workload. As a pilot develops skills and tasks become more automatic (Anderson, 1985), fewer attentional resources may be required to complete the task (Schneider & Shiffrin, 1977; Shiffrin & Schneider, 1977), resulting in a lowering of mental workload. Rasmussen (1986) developed the skill–rules–knowledge (SRK) framework that suggested that workload decreases as automaticity of the task increases (i.e., the move from knowledge-based reasoning to skill-based execution). If lack of practice causes someone to revert to knowledge-based reasoning for a previously learned skill, workload may increase. Thus if cognitive skill degradation is the same as a loss of skills, it is postulated that there should be a concomitant increase in mental workload when reverting to manual operation of a task after prolonged use of automation.

Cognitive skills may also be more vulnerable to decay if the operator has a lower level of expertise (Prophet, 1976). It is important to understand which cognitive skills are vulnerable to decay due to lack of practice in order to determine effective approaches to mitigate the loss of skills. Further research must be done to evaluate how pilots can best maintain their cognitive and physical flying skills (BASI, 1998; Gillen, 2008).

Much of the emerging automation being developed for the flight deck is designed for information support rather than aircraft control. Cognitive skill degradation has been identified as a potential issue in information automation systems (Archer, 2012; Casner et al., 2014; Dorneich et al., 2013; Hendrickson et al., 2006). Information automation is related to the management and presentation of relevant information to the flight crew (Abbott et al., 2013). Information automation encompasses all aspects of data collection (e.g., from sensors, databases, human input), processing (e.g., filtering, prediction from models, varying levels of abstraction), and presentation to the human operator(s) through any appropriate modality (e.g., visual, auditory, tactile; Billings, 1997; Nakamura, 2013). Information automation involves automating some of the cognitive elements of a task that a human operator would usually perform. Tasks that are affected include memory recall, calculations, comprehending, reasoning, prediction, situation assessment, making decisions, understanding alerts and warnings, predicting future states, and generating action plans (Anderson, 1982; Werfelman, 2015). These skills allow pilots to plan, to maintain SA at all times, to quickly assess new situations, and to make the best decision from the options available to them.

Some work has begun to collect empirical data on the effects of information systems on the retention of cognitive skills (Casner et al., 2014), but more is needed. Anecdotal evidence that this is a potential safety issue is available through reports on the Aviation Safety Reporting System (ASRS) as well as National Transportation Safety Board (NTSB) accident investigation reports. Measurement and analysis of the effects of information automation on cognitive performance are an important first step in understanding the root causes of these types of errors and in addressing them through mitigation recommendations.

To further evaluate cognitive skill degradation in information automation, the cognitive skills involved in flight planning were examined. Flight planning was chosen since it is one of the leading causal factors in general aviation accidents. Among the Federal Aviation Administration’s (FAA’s) list of ten most frequent causal factors for general aviation accidents are “inadequate preflight preparation and/or planning” and “mismanagement of fuel.” (Federal Aviation Administration, 2019) Between 2009 and 2018, there were 695 general aviation accidents that involved fuel mismanagement (starvation or complete exhaustion), with 86 fatalities (Aircraft Owners and Pilots Association, 2019). Increasingly the cognitive tasks of flight planning and flight monitoring are being automated (Dorneich et al., 2017). Flight planning incorporates various cognitive skills and can be done both manually and with the aid of automation. This research attempted to understand which tasks could suffer due to cognitive skill degradation, and the effects of cognitive skill degradation over time. Study 1 was an applied cognitive task analysis (ACTA) to identify the cognitive skills associated with flight planning, and pilots were asked to predict which cognitive skills might be most susceptible to skill degradation. Study 2 was an empirical evaluation to examine the effects of cognitive skill degradation over time through the use of flight planning automation.

Study 1: Applied Cognitive Task Analysis

Objectives

The objective of Study 1 was to better understand the demands on the cognitive skills needed during planning a flight (i.e., “cognitive demands”), what aspects of flight planning are susceptible to errors, and the cognitive skills required. A better understanding of cognitive skills required for flight planning was developed to aid in the prediction of which cognitive skills might degrade over time.

Methods

Analysis process

The primary goal of a cognitive task analysis (CTA) is to describe the tasks and plans required of a user to accomplish a specified goal, with a focus on illustrating the cognitive elements of tasks such as goal generation, decision-making, or judgment. Militello and Hutton (1998) developed the ACTA method by adapting CTA to be more streamlined and usable and to enable designers to evoke critical cognitive elements within a specific task from in-depth interviews with subject matter experts (SMEs; Seamster & Redding, 2017). ACTA uses various techniques to produce knowledge representations: hierarchical task diagram, knowledge audit, simulation interview table, and cognitive demands table. The representations complement each other and are intended to look at different aspects of cognitive skills. Based on the interviews, a list of cognitive skills was developed.

Flight planning tasks

Flight planning is the process of creating a flight plan, which describes a future flight. There are two main aspects of this process: calculation and compliance with FAA requirements. Calculation involves fuel requirements, distance from the origin to destination, and time en route. Calculating fuel involves determining the route, altitude, winds, and speed by optimizing fuel amount and time en route (Federal Aviation Administration, 2018; Tokadli, 2015). Fuel consumption is affected by variables such as winds, altitude, and weight on board. The pilot must take into account these factors to create an optimal plan.

Safety regulations require a minimal amount of fuel on board to take while flying and to account for any diversions. Pilots must take into consideration any notices for the airspace they are flying through (e.g., closures) while planning, airport conditions, and weather forecasts. Aircrafts must maintain a certain distance from the clouds and the ground when flying under visual flight rules, keep a visual reference to the terrain, and detrmine the altitude at which they fly.

Creating a fully optimized flight plan requires significant calculation, and automation can aid in this process. Calculations can be made using a manual device such as an E6-B and sectional chart. An E6-B is a modified circular slide ruler that functions as an analog calculating device for use in aviation. Alternatively, calculations can be made on an automated computer program.

Participants

The study consisted of six SMEs (five male, one female) with an average age of 37 years (range: 21–62). The participants consisted of two professional pilots, three flight instructors, and one experienced general aviation pilot. Pilots had an average of 1100 flight hr (range: 200–2800). Airplanes that the participants have flown include: Beechcraft Bonanza, Cessna 172, Cessna 182, Cirrus SR22 TN, F-15, F-16, Piper Cherokee, Piper Comanche, Piper PA28R, Piper PA44, Piper Warrior, RU12, and Socata TBM. All participants held a private pilot license; four also held a commercial pilot license and one held a military and air transport license. When asked how familiar each was with the E6-B, four participants answered “very familiar, I use it frequently” and two participants answered “slightly familiar, I use it occasionally.” Five participants also noted that they used other online tools such as SkyVector, ForeFlight, and iFlightPlanner to assist in the flight planning process.

Procedure

Participants signed a consent form, completed a basic demographic survey, were briefed, and given the opportunity to ask questions. Data collection was split into two phases. The first phase was an interview that collected data for the task diagram and knowledge audit. The second phase was a simulation interview that introduced a challenging scenario for pilots to complete to elicit information about the demands on the cognitive skills associated with the task. The entire study was audio-recorded.

In the first phase, pilots were interviewed and asked a series of questions (Table 1) in order to elicit a step-by-step procedure for flight planning. Follow-on prompts were used to probe more deeply to understand the knowledge, skills, and strategies needed for each step. The interview asks for specific examples and events in terms of critical cues and strategies of decision-making. For instance, if their responses suggested decision-making (e.g., use of verbs such as “think,” “decide,” and “choose”), then they were further prompted to talk about the criteria for each decision. Finally, a discussion ensues about the potential errors that a novice, less experienced person might make in similar situations (Militello & Hutton, 1998). Militello and Hutton (1998) found that the knowledge audit was insufficient when used alone without a deep understanding of the task itself. Hence ACTA is used with experts who have that deep understanding of the task, and those experts are relied upon to identify novice errors and issues. Table 1 presents the interview probes or questions (in the order they were presented to participants) and the rationale based on the ACTA methodology (Militello & Hutton, 1998). Questions were altered slightly to apply to aviation.

Table 1

Interview Question Rationale Based on Methodology From Militello and Hutton (1998)

Question	Probe Category	Rationale
Can you walk me through the process of planning for a flight?	Big picture	Experts are able to assess a situation by understanding all elements of it while novices may only see small elements rather than the big picture. Seeing the big picture allows the expert to see how elements affect each other.
How do you plan for future events?	Past and future	Experts are able to understand how a situation arose and what possible outcomes are ahead. This allows them to address problems before they occur.
What do you take into consideration when planning a flight?	Big picture	Experts are able to assess a situation by understanding all elements of it while novices may only see small elements. Seeing the big picture allows the expert to see how elements affect each other.
How do you pick the optimal route?	Job smarts	Experts are able to work efficiently and not waste time or resources.
How does the weather affect the fuel you carry?	Noticing	Experts are able to look at patterns and cues to develop strategies which a novice may not see.
Is there a time when you walked into the middle of a situation and knew exactly how things got there and where they were headed?	Past and future	Experts are able to understand how a situation arose and what possible outcomes are ahead. This allows them to address problems before they occur.
Can you give me an example of what is important about the Big Picture for this task? What are the major elements you have to know and keep track of?	Big picture	Experts are able to assess a situation by understanding all elements of it while novices may only see small elements. Seeing the big picture allows the expert to see how elements affect each other.
Have you had experiences where part of flight planning just “popped” out at you; where you noticed things that others usually do not catch? What is an example?	Noticing	Experts are able to look at patterns and cues to develop strategies which a novice may not see.
When you do this task, are there ways of working smart or accomplishing more with less—that you have found especially useful?	Job smarts	Experts are able to work efficiently and not waste time or resources.
Can you think of an example when you have improvised in this task or noticed an opportunity to do something better?	Opportunities / Improvising	Experts can improvise based on previous experience with comfort, they are also able to see opportunities to be more efficient and take them.
Can you think of a time when you realized that you would need to change the way you were performing in order to get the job done?	Self-monitoring	Experts are able to assess their own performance and make necessary adjustments. Novices may not know how to improve or are not aware of their performance.
Can you describe an instance when you spotted a deviation from the norm, or knew something was amiss?	Anomalies	Experts can easily spot abnormal events or deviations while novices are unaware of what is atypical. Experts are able to notice when something that was supposed to happen, doesn’t.
Have there been times when the equipment (e.g., flight planning software, information services, etc.) pointed in one direction, but your own judgment told you to do something else? Or when you had to rely on experience to avoid being led astray by the equipment?	Equipment Difficulties	Novices typically believe what the equipment outputs and do not know when to be skeptical of an error.

In the second phase (simulation interview), pilots filled out the FAA 7233–1 form, which involves basic flight planning information. This includes: type, aircraft identification, departure point, destination, route of flight, and fuel on board. They were also given an E6-B (Aviation Supplies & Academics, 2012) and a sectional chart of the Ames, IA to Minneapolis, MN area, which included weather information from SkyVector.com. Pilots could use a dry erase marker to draw on the sectional as desired. They were asked to manually calculate various elements such as speed, heading, time, and fuel requirements under the following scenario:

You will do the flight planning stage for a flight from Ames to Minneapolis. You are flying a Cessna 172 with the fuel capacity of 42 gal. The range of the aircraft is 435 nautical miles, cruise speed is 115 knots, and the direct distance between KAMW and KMSP is 174 miles. Please fill out the FAA 7233-1 form and plan each step out loud. All of the information at each waypoint is provided as well as a map with weather information, and an E6-B flight computer. Afterward, I am going to ask you a series of questions about how you would approach this situation.

During the scenario, pilots were asked to speak out loud any decision points or observation points they encountered. Following the scenario, pilots were asked a series of questions (Table 2) to elicit deeper information.

Table 2

Simulation Interview Question Rationale Based on Methodology From Militello and Hutton (1998), Altered Slightly to Apply to Aviation

Question	Rationale
As the pilot in this scenario, what actions, if any, would you take at this point in time?	To understand the SME’s situation assessment of the scenario presented to them.
What do you think is going on here? What is your assessment of the situation at this point in time?	To understand what actions the SME would take based on the information from the scenario.
What pieces of information led you to this situation assessment and these actions?	To understand what critical cues lead to decision making.
What errors would an inexperienced person be likely to make in this situation?	To understand what potential errors could occur and the differences between novices and expert decision making.

Note. SME = subject matter expert.

Knowledge representations

Interview transcripts were separated by interview question and input into a spreadsheet to compare between pilots. Interview responses were reviewed to identify themes, cues, and patterns, as well as create a synthesized/integrated narrative. Four knowledge representations were produced:

Hierarchical task diagram provides a general overview of a task and identifies difficult cognitive elements of the task.

Knowledge audit describes each task and subtask in terms of specific examples in the SME’s experience, to uncover different aspects of expertise. The SME identifies the cues and strategies used along with potential difficulties for novices.

Simulation interview table identifies key decision points, actions, critical cues, and sources for error (i.e., potential errors).

Cognitive demands table consolidates and synthesizes the data from various interviews to describe what makes certain cognitive elements difficult, sources of (common) errors, and strategies used to complete specific tasks.

Cognitive skills table

A table of cognitive skills was developed based on the responses from participants during the ACTA interview. Any mention of cognitive elements of tasks were collected, consolidated, and sorted into a list of “cognitive skills.” Definitions for each skill were derived from literature and dictionaries. Examples were taken from the interview responses. During the interview, pilots also indicated which tasks were difficult for novice pilots.

Limitations

ACTA does not always capture noncognitive attributes needed in order to fully understand a task. This could include access to resources and interpersonal relationships. Furthermore, it is assumed that experts can accurately articulate the difficulties of a novice and remember the performance capacities and capabilities of a beginner. ACTA assumes that the difficult parts of a task can be revealed by understanding expert-novice differences. However, some tasks can be difficult for experts as well. For example, some tasks may necessitate teamwork or may involve nonroutine scenarios that may be equally difficult for experts and novices.

Results

Figure 1 presents the hierarchical task diagram. The task was broken into eight main steps with substeps to complete the task. This flight plan procedure is applicable to a general aviation Visual Flight Rules (VFR) flight. Table 3 presents the result of the knowledge audit. Table 4 describes the simulation interview table. Table 5 describes the cognitive demands table. Table 6 lists cognitive skills required during flight planning.

Figure 1

Hierarchical task analysis for planning a general aviation VFR flight.

Table 3

Knowledge Audit

Aspects of Expertise	Cue and Strategies	Why Difficult?
Past and Future Need to plan a new route; gathering information from various sources to understand the situation	Roughly map flight to determine waypoints. Talk to other pilots with more experience. Watch videos of approaches and landings to familiarize themselves with destination airport. Understand jargon and communication with Air Traffic Control (ATC).	Novices less familiar with routes. Novices have difficulty planning for change.
Big Picture Assessing route and environmental conditions to determine if there is a viable flight plan	Determine time and distance between waypoints. Review weather along the route. Review NOTAMs^a for origin and destination. Review turbulence reports. Review Temporary Flight Restrictions (TFRs). Look at weather terminal forecast. Review runway conditions. Calculate fuel requirements. Calculate speed, heading, and flight time. Calculate weight and balance.	“Get-there-itis”: determination of pilot to get to destination, even when flying conditions are dangerous. External pressures (external influences that create a sense of pressure to complete a flight).
Noticing Reading NOTAMs for destination airports	Read NOTAMs first so you do not plan the entire flight to a closed airport. Distribute the workload with another pilot if possible. Work efficiently so weather does not change by the time the flight is filed.	Unfamiliarity with NOTAMs codes; high workload; understanding how often to recheck.
Noticing Changes in the weather	Review the direction of the winds to look at headwinds and tailwinds. Review NOTAMs for origin and destination. Review TFRs.	Winds change at various altitudes; understanding how often to recheck.
Job Smarts Work efficiently	Starts with the big picture and scale down. Use electronic apps. Distribute the workload with another pilot.	Novices may take too long to calculate, risking outdated environment information. Planning things out of order (i.e., determining waypoints before looking at weather report).
Opportunities/Improvising Navigation tools and techniques	Use electronic apps to help calculate variables (i.e., weight and balance, fuel, winds, etc.). Double-check inputs if calculations seem off.	Noticing miscalculations when workload is high; do not have a good mental model of the process.
Self-Monitoring Adjusting and streamlining	Roughly plan ahead days before. Understand the performance characteristics of your aircraft. Expose yourself to new information (i.e., talk to other pilots, read articles, etc.). Follow a procedure for how to plan.	Comfortable doing things the same way and not learning how to improve.
Anomalies Calculations seem off	Recheck input numbers for calculations. Ability to estimate outputs. Expectation of what is correct.	Novices not being aware of what is right.
Equipment Difficulties Equipment malfunctions	Double-check information with ATC. Pay attention to the equipment readings. Be prepared for things to malfunction. Understand emergency procedures.	Understanding how the equipment works to see if something is wrong. Novices may hesitate to question ATC.

^aNOTAM = Notice to Airmen.

Table 4

Simulation Interview Task Results

Events	Actions	Assessment	Critical Cues	Sources for Error
Construct flight plan	Determine origin, destination, and alternate airport	Enough fuel, runway conditions sufficient	Airport is closed or open	Planning flight without checking if the airport is viable
	Develop initial route	Waypoints close enough from one another. Altitude viable to fly at.	Check aircraft manual for aircraft performances	Novice may try to stay at one altitude entire flight. Developing an inefficient route and wasting fuel.
	Check weather forecast	Visibility above minimum. Weather conflicts. High winds.	Visibility >300 ft. for day. Visibility >500 ft. for night. Icing en route.	Believing that weather forecast won’t change. Believing the weather at station is same for any altitude.
	Check weather conditions at airports	Hazardous weather that could affect takeoff and landing	Moisture or icing on runway	Not checking terminal forecast
	Check NOTAMs^a	Closures. Runway conditions.	Airport open/closed. Runway open/closed. Lighting on runway is broken or working. Hazardous conditions exist.	Novices may not be familiar with all of NOTAM codes
	Check TFRs	Restricted airspaces	Airspace open/closed	Novices may not check TFRs
	Calculate speed, heading, flight time, and total fuel	Fuel stop required. Tailwind or headwind. Time en route within level of comfort.	Aircraft can hold fuel needed	Not taking into account fuel use for takeoff and landing
	Calculate weight and balance	Weight and balance in line with aircraft specifications.	Check aircraft manual	Incorrectly inputting the values for weight and balance
File flight plan	File with ATC	Flight ok to perform	Approval from ATC	Taking so long to plan a flight that weather conditions have changed
Taxi runway	Obtain updated weather	Need to replan	Weather has changed drastically or not	Overconfident that changed weather will not affect safety
Taxi runway	Request flight following	Flight planning provided	Approval from ATC

^aNOTAM = Notice to Airmen

Table 5

Cognitive Demands

Difficult Cognitive Element	Why Difficult?	Sources of Errors	Cues and Strategies Used
Knowing how to deal with uncertainty of future	Unfamiliarity with the route	Overconfidence. Wasted time.	Speak with experienced pilots. Watch videos. Look at terminal forecasts. Train for off-nominal situations.
Noticing when something is not right	This skill comes with experience	Trusting automation fully. Not understanding how the automation is calculating. Not understanding NOTAM^a codes. Not being able to estimate what the right answer should be.	Learn how to do calculations by hand first and practice it. Double-check what inputs were used. Be able to assess the situation. Double-check the information given. Speak with experienced pilots.
Assessing the viability of a flight	Various elements come into play, and it is more difficult with less experience and familiarity	“Get-there-itis.” External pressures. Misreading NOTAMs. Wasting time.	Start by looking at weather report. Do not let external pressures compromise safety. Understand NOTAM codes.
Synthesizing the preflight information	Have to consolidate various data to create a bigger picture	Misinterpreting information	Review with a more experienced pilot. Obtain information from manuals or ground operators.

^aNOTAM = Notice to Airmen

Table 6

Cognitive Skills in Flight Planning With Definitions Based on Merriam-Webster (2018) and ACTA Interviews

Cognitive Skill	Definition	Example
Calculating	To determine outputs by mathematical processes	Calculating the required fuel for the flight
Estimating	To form an approximate judgment of what the calculation should be	Estimating what the required fuel will be prior to exact calculation
Noticing	The condition of being warned or notified of something that could hinder the flight plan	A NOTAM^a states that the destination airport has a runway which is closed
Organizing	To arrange all information by systematic planning	Laying out all materials needed to efficiently plan
Controlled processing	To process all variable which requires substantial mental resources	Look at the big picture of all the information to formulate a plan
Reasoning	To comprehend, infer, or think in a rational manner to ensure that all information seems accurate	A headwind will require more fuel
Problem-Solving	To find a solution to a problem	A storm is approaching from the West and the pilot must determine how to avoid it
Scanning	To examine given information systematically in order to obtain data	Overview all information to see if anything pops out that needs attention
Predicting	To foresee issues and deal with in advance; to declare or indicate in advance	Knowing that the weather will be more variable in the summer months
Recognizing	To perceive to be something based on frequency or experience, and to act based on relevant cues, expectancies, and goals.	Recognizing that the current situation is “close enough” to a pattern-match to a known situation to apply a known flight planning solution (e.g., diversion), with perhaps some improvisation.
Communicating	To convey knowledge or information about an idea	Talking with the control tower
Prioritizing	To determine order based on importance	Looking at the weather report before calculating fuel requirements

^aNOTAM = Notice to Airmen

Implications

Based on the interviews, the skills most difficult for novices initially were those that involved calculation and estimation. Calculating different variables can be difficult for novices, but can improve with practice over time. From the interviews, pilots mentioned that cognitive skills such as noticing when something “does not look right” are gained from experience and prone to error for novices, especially when under high workload. Estimation of approximate values supports noticing. Both skills may be vulnerable to skill decay if not practiced over time. Minotra and Feigh (2017) highlight the role of experience and the role of task saturation or workload in tasks that involve noticing or judging; with experience, mental workload is reduced and tasks such as noticing become easier.

Calculation includes heading, speed, time en route, and fuel requirements. Calculation is a difficult skill that relies on working memory and requires the pilot to synthesize various variables, then properly use them in order to receive an accurate output. Pilots noted that inexperienced pilots are prone to mistakes, such as incorrect inputting values into automation or using a provided online template for aircraft specifications that contain errors. For example, incorrect distance calculations impact the fuel calculations, which can be dangerous in flight if not correct. Pilots reported that they felt it was vital to learn how to do these tasks manually before graduating to automation aids. This was due to the need to develop estimation and noticing skills through experience. While calculation skills in and of themselves may not be necessary since the automation will calculate values for them, calculations skills may be important inasmuch as they support the development of estimation and noticing skills through experience.

Estimation involves determining an approximate value for the output, and checking if the calculation matches the estimation. If the calculation does not seem right, the pilot needs to recalculate to ensure it is correct. Pilots preferred using automated methods to perform flight planning due to its efficiency and reliability. Pilots preferred to use automation for exact calculations and manually estimate approximate values. It was stressed how automation can make things efficient; however, a good pilot knows what the automation is doing and can check the output to see if the numbers approximately match their expectations. Checking the outputs of automation will lead to a better calibration of trust in the automation. Inexperienced pilots may not question the automations outputs. For example, if the input data were incorrect or incorrectly inputted, the calculation can give an incorrect output, which is not noticed by the pilot. Novices can fail to verify information given by automation if they are overloaded with tasks (Minotra & Feigh, 2017; Mosier et al., 2018). Without practice, pilots may lose the skill of doing a mental check to determine if the numbers seem reasonable and proceed to use them regardless. Duration between instances where the skill is used before there is a skill loss is less known and should be investigated further to see how individuals lose skills over times of nonuse.

Study 2: Empirical Evalution

Objectives

Based on the ACTA, it was hypothesized that within flight planning there are two skills types particularly vulnerable to decay: calculation and estimation. Pilots felt that without the consistent practice of these skills, they may be susceptible to degradation. This lead to hypothesis H1:

H1: Repeated use of an automation aid is expected to result in larger degradation in flight planning performance than the repeated use of manual flight planning over time.

Skill and workload are intrinsically linked. Skill can be generally defined as the “efficient use of appropriate behavior which develops with experience” (Bainbridge, 1989, p. 87). As the operator gains skills, they perform the task with increasing efficiency and minimum of effort. Thus, as skill increases, workload associated with those cognitive tasks should decrease. This leads to hypothesis H2:

H2: Reliance upon automation aids will lead to higher workload and completion time when the user is required to return to manual calculation and estimation skills. This is in contrast to participants who only use manual flight planning.

Methods

Participants

A total of 59 undergraduate students from a large Midwestern university served as participants initially. No participants stated that they had previous experience with flight planning. Students were chosen as the study group to ensure that everyone started with no prior knowledge of the task or aviation that would confound the results with varying levels of aviation experience. This allowed participants to be trained to criteria and start the experimental trials with a common level of task knowledge and experience. The experiment required participants to return every 2 weeks for 9 weeks. A total of 46 participants completed all five trials (26 male, 20 female), with an average age of 19.7 (range: 18–27). The cumulative attrition rate was 22%. Only the 46 remaining participants’ data were analyzed. The experimental plan and preliminary results were presented in Volz et al. (2016).

Tasks/scenarios

In order to plan a flight, participants were asked to calculate the following elements of a flight segment: heading, ground speed, flight distance, time en route, fuel consumption, and gallons burned per hour. Definitions and problem statement examples are shown in Table 7. Participants were asked the same problem statements each trial with different values for each variable.

Table 7

Task Descriptions and Examples

Element Question	Description	Problem Example
Heading	Direction the plane is pointed (must account for wind).	The weather report indicates that there are winds from 240° at 38 knots (KTS). Your course is 300° and your aircraft has a true airspeed of 165 KTS. Calculate the true heading and ground speed.
Ground speed	Speed that the plane is flying relative to the ground.
Flight distance	Distance between the two endpoints of the flight.	The ground speed is 110 knots (KTS); your trip will take 1 hr and 40 min. Calculate the distance.
Time en route	Length of time the plane is in the air.	The weather report has changed and the ground speed is now 125 knots (KTS); the distance of your trip will be 524 nautical miles. Calculate the time that your trip will take.
Fuel consumption	Amount of fuel used by the plane during the flight.	Your aircraft burns an average of 7.5 gallons per hour; your trip will take 3 hr and 20 min. Calculate total fuel used.
Gallons burned per hour	Rate at which the fuel is being used up.	Your trip will use 62 gallons of fuel, and the trip will take 4 hr and 50 min. Calculate gallons burned per hour.

Independent variables and experimental design

At the beginning of the study, participants were randomly placed into the three experimental groups: manual, alternating, and automation. The manual group used an E6-B for every experiment trial. The alternating group switched every trial between an E6-B and an E6-B emulator (myE6B.com, Aviation Mobile Apps LLC, Cape Canaveral, FL) on an HP Stream 8 32 GB Windows 8.1 4G 8 inch Tablet. The alternating group was included to determine if participants received manual practice at longer intervals, it might be possible to see if skills recovered or stabilized with intermittent practice. The automation group used an E6-B for the first trial, the automated E6-B for the three consecutive trials, and back to the regular E6-B during the last trial. Trial 1 was a baseline manual task for all groups. Trials 2–4 were practice trials that differed between groups. Trial 5 was the criterion test where all groups conducted the task manually. Of the 46 participants who completed the study, 18 were in the manual group, 12 in the alternating group, and 16 in the automation group. The between-subjects experimental design is shown in Figure 2.

Figure 2

Experimental design.

Dependent variables/metrics

The dependent variables of performance, completion time, and workload were measured after every trial. Performance was determined by the percent correct (on six questions) after each scenario, where the participant needed to calculate heading, ground speed, distance, and total fuel used. Answers were considered correct if they were within one unit of the last significant digit (e.g., if the correct answer was 1.3, then acceptable answers were in the range 1.2–1.4). Subjective workload after each trial was assessed with the NASA Task Load Index (NASA-TLX), by calculating the average of the six subscales (mental demand, physical demand, temporal demand, performance, effort, frustration), where each was scored on a 0–20 scale (Hart & Staveland, 1988). Completion time was calculated from when the participant received the scenario and when the final question was answered.

Procedure

After signing a consent form, participants completed a demographic survey, were briefed, and were trained on NASA-TLX. An initial training session was given to all the participants on how to use an E6-B via video, instruction, and practice. Next, the users took a practice test to confirm their understanding. If they did not answer all questions correctly, an experimenter would work with the participant individually to identify and correct any errors. Training continued until they were able to score 100% on the practice test.

Once participants had completed the practice test with no errors, they began the trials. In every trial, participants completed two scenarios consisting of three questions each. Throughout the experiment, the only alterations made were the numbers given in the problem statements. The difficulty in the questions remained the same. Immediate feedback after each trial was intentionally not provided to participants because the experimenters did not want to provide feedback or interfere with the skill level of participants. After the scenarios, they were instructed to fill out a NASA-TLX questionnaire.

Limitations

It is possible that no feedback after each trial would lead to lower performance because participants did not know whether their approach was correct or not. However, they were trained to criteria before the trials began. Without feedback, the participants would not have the natural learning associated with the task that happens in the real world. In the real world, task pilots would likely realize a mistake when trying to enter or execute the flight plan.

Data analysis plan

Each of the dependent variables were compared between groups and trials. Analysis of variance (ANOVA) was used to calculate the main and interaction effects of the three groups over the five trials. Post-hoc analysis was conducted by using Tukey’s test for pairwise comparisons. Satterthwaite approximation t-tests were used for comparing results between Trial 1 and Trial 5 within each group, and for comparing the change from Trial 1 and Trial 5 between groups. The results are reported as significant for α <.05. Error bars represent standard error between participants within a group.

Results

Performance

Figure 3 illustrates the average performance of the three groups over five trials. The main effect of group was significant (F(2, 62.9) = 13.99, p < .001). The main effect of trial was significant (F(4, 171) = 39.17, p < .001). The interaction was significant (F(8, 171) = 18.81, p < .001).

Figure 3

Average performance of participants on flight task over five trials.

Figure 4 illustrates the average performance for each group between the first and last trials. For the manual group, there was a significant (t(17) = 15.6, p = .010, d = 1.08) decrease in performance from Trial 1 (M = .70, SE = .04) to Trial 5 (M = .50, SE = .06). For the alternating group, there was a significant (t(11) = 35.7, p < .001, d = 1.23) decrease in performance from Trial 1 (M = .83, SE = .04) to Trial 5 (M = .47, SE = .08). For the automation group, there was a significant (t(15) = 32.4, p < .001, d = 1.69) decrease in performance from Trial 1 (M = .77, SE = .05) to Trial 5 (M = .37, SE = .07).

Figure 4

Average performance of participants on flight task for each group for the first and last trial; p = values are presented for significant difference between Trial 1 and Trial 5.

Figure 5 illustrates the average change in performance for each group between Trial 1 and Trial 5. The main effect of group was significant for the change in performance, F(2, 2) = 3.33, p = .045. The change in performance was significantly (p = .045, d = .79) greater in the automated group (M = −.40, SE = .06) than the manual group (M = −.20, SE = .06). No other pairwise comparisons resulted in significant differences.

Figure 5

Average performance of participants on flight task for the difference in each group between the first and last trial; there is a significant difference between the manual and automation groups.

Average workload

Figure 6 illustrates the average workload of the three groups over five trials. The main effect of group was not significant, F(2, 58.4) = 1.56, p = .22. The main effect of trial was significant, F(4, 170) = 40.76, p < .001. The interaction was significant, F(8, 173) = 35.63, p < .001.

Figure 6

Average workload of participants after flight task over five trials.

Figure 7 illustrates the average workload for each group between the first and last trial. For the manual group, there was no significant difference in average workload from Trial 1 to Trial 5, t(17) = 1.38, p = .26. For the alternating group, there was no significant difference in average workload from Trial 1 to Trial 5, t(11) = .33, p = .57. For the automation group, there was a significant (t(15) = −10.0, p = .006, d = −.82) increase in average workload from Trial 1 (M = 4.4, SE = .51) to Trial 5 (M = 5.74, SE = .48).

Figure 7

Average workload of participants on flight task for each group for the first and last trial; there was a significant difference between Trial 1 and Trial 5 for the automation group.

Comparing the change in average workload from Trial 1 to Trial 5 between groups

Figure 8 illustrates the average workload for each group between the first and last trial. The main effect of group was significant for the change in average workload, F(2, 2) = 4.73, p = .014. The change in average workload was significantly (p = .010, d = 1.01) greater in the automated group (M = 1.29, SE = .44) than the manual group (M = −.55, SE = .41). No other comparisons resulted in significant results.

Figure 8

Average workload of participants on flight task for the difference in each group between the first and last trial; there was a significant difference between the manual and automation groups.

Completion time

Figure 9 illustrates the average performance of the three groups over five trials. The main effect of group was significant, F(2, 70.6) = 6.49, p = .003. The main effect of trial was significant, F(4, 174) = 6.31, p < .001. The interaction was significant, F(8, 174) = 16.44, p < .001.

Figure 9

Average completion time of participants after flight task over five trials.

Figure 10 illustrates the average completion time for each group between the first and last trial. For the manual group, there was no significant difference in completion time from Trial 1 to Trial 5, t(17) = 5.13, p = .037, from Trial 1 (M = 8.9, SE = .71) to Trial 5 (M = 10.1, SE = 1.01). For the alternating group, there was no significant difference in completion time from Trial 1 to Trial 5, t(11) = .87, p = .36. For the automation group, there was no significant difference in completion time from Trial 1 to Trial 5, t(15) = 1.84, p = .20.

Figure 10

Average completion time of participants on flight task for each group for the first and last trial.

Figure 11 illustrates the average change in completion time for each group between the last and first trial. The main effect of group was significant for the change in completion time, F(2, 2) = 3.29, p = .047. The change in average completion time was not significantly (p = .059, d = .99) greater in the automated group (M = 1.19, SE = .78) than the manual group (M = −1.33, SE = .73). No other comparisons resulted in significant results.

Figure 11

Average completion time of participants on the flight task for the difference of the last trial minus the first trial.

Discussion

The study investigated the effects of cognitive skill degradation through the use of automation. H1 stated that the use of an automation aid is expected to result in larger degradation in flight planning performance than the repeated use of manual flight planning over time. The hypothesis was supported. Performance degraded in all three conditions: manual, alternating, and automation. However, the degree of performance decay was larger in the automation group than the manual group, when comparing the difference between Trial 1 and Trial 5 across the two groups (Figure 5). Based on the ACTA, it was hypothesized that within flight planning, there are two skills types particularly vulnerable to decay: calculation and estimation. The performance decay may be indicative of a cognitive decay in these skills. The performance decay involved errors of calculation, and poor estimation skills may be one reason participants were not able to detect these errors.

Additionally, one can investigate the drop in performance between the first time and the second time a group encountered the manual method of flight planning. In the manual group, Trial 1 and Trial 2 are the first two times they encountered the manual methods of flight planning (see the round markers of Figure 3). The manual group’s performance dropped from Trial 1 to Trial 2, but stayed level in subsequent trials. This initial drop could be an artifact of the experimental design, such that the training criteria of 100% on the training trials was insufficient to ensure that participants had learned the task well enough to retain the level of performance from Trial 1 to Trial 2. However, performance stayed approximately level after that through Trials 2–5. While there was no further degradation in performance, the repeated manual trials did not improve performance either. Thus, repeated exposure to the manual method helped to retain, but not improve, on the task performance. This could be due to the lack of feedback after each trial, which could have helped skill improvement over the practice trials.

Another analysis of interest is to compare the first two manual trials in each practice group (see the round markers of Figure 3). The manual group’s second manual trial was in Trial 2 (i.e., 2 weeks after Trial 1). The alternating groups’ second manual trial was in Trial 3 (i.e., 4 weeks after Trial 1). The automation group’s second manual trial was in Trial 5 (8 weeks after Trial 1). In each case, the drop-off in performance between the first and second manual trial increased with increased time between manual trials. Thus, the longer time between manual trials, the larger the decrease in performance, where the intervening trials with the automation aid did not stem the drop-off in performance. A similar trend can be seen for workload.

H2 stated that reliance upon automation aids will lead to higher workload and completion time when the user is required to manually plan the flight. The hypothesis was partially supported. While the manual group saw their workload decrease from Trial 1 to Trial 5, the automated group saw their workload rise from Trial 1 to Trial 5. A similar result was seen for the completion time, but the difference did not rise to the level of significance.

The results of this study show that the persistent use of an automation aid presented the highest level of skill degradation between the first and last trial. All three groups decreased in performance when comparing Trial 1 to Trial 5. Specifically, the manual group displayed the least degradation of performance. The alternating group demonstrated moderate degradation of performance. The automation group showed the highest degradation of performance.

Although all three groups experienced degraded results of performance over time, the manual group had the least amount of degradation whilst the automation group showed the highest degradation. This indicates that reinforcing the practice of a task manually mitigated skill degradation.

The learning effect is a possible explanation for the manual and alternating groups having less degradation in comparison to the automation group. The additional trials where subjects used the E6-B to calculate the answers in the manual and alternating groups could act as additional practice where participants were retrieving the skills to use the E6-B, therefore enhancing their learning. Attempting to recall information has been shown to enhance learning (Kornell & Vaughn, 2016; Rowland, 2014).

Reflecting upon workload, as more time elapsed between using the manual method, average workload increased. Using the automated method provides participants with a lower workload whilst using the aid; however, relying on automation increases workload when switching back to the manual method. For the alternating group, their workload was approximately the same between the first trial and last trial, which indicated that a mixture of using the manual method with the automated aid was helpful in not raising average workload.

The average completion time did not result in any significant differences between groups (Figure 11); however, trends can be observed. When the automated group switched back to the manual method, it took longer for them to complete the task than the other groups. This could be due to a long time in between conducting the task with the manual E6-B. This may indicate that the use of automation aids is insufficient as practice for a task to keep manual skills current. The manual group took longer the second time they performed the task manually (Trial 2), and then continued to reduce completion time each subsequent trial.

Calculation and estimation skills are thought to be vulnerable to decay because they require time, experience, and working memory (Wan & Huon, 2005). Working memory for tasks can improve with experience (Ericsson & Kintsch, 1995). Participants were not consistently practicing these skills; therefore, they were susceptible to degradation. The automation aid was not sufficient as practice for the alternative or automation groups when they were tested on the manual method.

Conclusion

The first study used an ACTA to identify the demands of the cognitive skills needed in flight planning. The level of difficulty each skill has was identified for novices as well as experts, and what errors may result. The cognitive skills identified were: calculating, estimating, noticing, organizing, processing, reasoning, problem-solving, scanning, predicting, recognizing, communicating, and prioritizing. In particular, pilots identified calculation and estimation as skills that are particularly vulnerable to degradation.

The second study was an empirical evaluation designed to infer the presence and magnitude of cognitive skills degradation as a result of differing levels of use of automation. It focused on how time and automation aids affected the performance, workload, and completion time of calculation and estimation skills of flight planning. Expert pilots in Study 1 identified the level of difficulty of each skill for novices and predicted that novices would suffer from loss of calculation and estimation skills. The results of Study 2 support this prediction. The study found that automation as an aid did not help in terms of practice. It also showed that after a period of nonuse for the manual method, completion time and workload increased. In order to mitigate the degradation of these skills, the manual method must be practiced.

Taken together, the studies imply that manual practice of flight planning tasks are important to maintain the cognitive skills of calculation and estimation. Pilots said they preferred to use automation for calculation and do estimation manually. While calculation skills in and of themselves may not be necessary since the automation will calculate values for them, calculation skills may be important inasmuch as they support the development of estimation and noticing skills through experience. Computational estimation skills have become more important as advances in technology requires users to determine the reasonableness of automation outputs (Reys & Bestgen, 1981). As other domains (e.g., semiautonomous driving, beyond earth orbit crewed spaceflight) incorporate increasingly capable information automation, they too may want to consider how much manual task practice is necessary to maintain the cognitive skills necessary to effectively monitor or team with automation.

The participants in Study 2 were students, to better approximate novice users. However, further work should investigate skill degradation with experts. Additionally, further studies need to establish how to mitigate skill degradation. Measurement and analysis of the effects of information automation on cognitive performance are an important first step in understanding the root causes of these types of errors and in addressing them through mitigation recommendations that should be considered during the design of information automation systems. These issues should be addressed in future work to expand understanding of the long-term effects of cognitive skill degradation.

Footnotes

Acknowledgment

This work was supported in part by the Undergraduate Research Assistant (URA) program in the IMSE department. Special thanks to the URA program coordinators Devna Popejoy-Sheriff and Leslie Potter. We would like to thank Rachel Dudley and Euijung Yang for guidance. Thank you to the URAs Elizabeth Lynch for her aid in developing test scenarios, Maria Dropps for her help running the experiment, and Nicole Kittleson for data transcription. Jeremy Hadler contributed as a statistics consultant in the data analysis portion. KV led the development of both experiments, conducted the experiments, analyzed the data, and wrote the first draft of the manuscript. MD assisted in the development of the experiment methods, analysis and interpretation of data, and the writing/revising of the manuscript.

ORCID iD

Michael C. Dorneich

Author Biographies

Katherine M. Volz is a human factors engineer at The Boeing Company. She earned her BS (2017) and MS (2018) in industrial and manufacturing system engineering at Iowa State University. Her work focuses on the application of principles of human capabilities and constraints, including ergonomics, usability evaluation, user-centered design, physical and cognitive interaction design, and statistical and human error analysis.

Michael C. Dorneich is an associate professor at Iowa State University. He earned his PhD at the University of Illinois at Urbana-Champaign. His research interests focus on creating joint adaptive human-machine systems that enable people to be effective in the complex and often stressful environments found in aviation, military, robotic, and space applications. He holds 28 U.S. and international patents. He has authored over 170 journal articles, book chapters, and refereed conference proceedings papers.

References

Abbott

McKenney

Railsback

. (2013). Operational use of flight path management systems. Final report of the Performance-based operations Aviation Rulemaking Committee (PARC)/Commercial Aviation Safety Team Flight Deck Automation Working Group (CAST).

Aircraft Owners and Pilots Association . (2019). Accident analysis: Fuel mismanagement. AOPA. https://www.aopa.org/Training-and-Safety/Air-Safety-Institute/Accident-Analysis/Fuel-Mismanagement

Anderson

J. R

. (1982). Acquisition of cognitive skill. Psychological Review, 89(4), 369–406.doi:10.1037/0033-295X.89.4.369

Anderson

J. R

. (1985). Cognitive psychology and its implications. WH Freeman/Times Books/Henry Holt & Co.

Archer

. (2012). Effects of automation in the aircraft cockpit environment: Skill degradation, situation awareness, workload. Purdue University.

Argote

Beckman

S. L.

Epple

. (1990). The persistence and transfer of learning in industrial settings. In Klein

D. A.

(Ed.),The strategic management of intellectual capital (pp. 189–209).

Bainbridge

. (1989). Development of skill, reduction of workload. In (L. Bainbridge & S.A. Ruiz Quintanilla, Eds. Trans.). Developing Skills with Information Technology (pp. 87–116). Wiley.

BASI . (1998). Advanced aircraft technology safety survey report. US department of transportation. D.o.T.a.R.D.B.o.A.S.I. (Australia), Bureau of air safety investigation (BASI).

Billings

C. E

. (1991). Human-centered aircraft automation: A concept and guidelines. National Aeronautics and Space Administration, Ames Research Center.

10.

Billings

C. E

. (1997). Aviation automation: The search for a human-centered approach. Lawrence Erlbaum Associates.

11.

Casner

S. M.

Geven

R. W.

Recker

M. P.

Schooler

J. W

. (2014). The retention of manual flying skills in the automated cockpit. Human Factors, 56(8), 1506–1516.doi:10.1177/0018720814535628

http://www.ncbi.nlm.nih.gov/pubmed/25509828

12.

Dorneich

M. C.

Dudley

Letsu-Dake

Rogers

Whitlow

S. D.

Dillard

M. C.

Nelson

. (2017). Interaction of automation visibility and information quality in flight Deck information automation. IEEE Transactions on Human-Machine Systems, 47(6), 915–926.doi:10.1109/THMS.2017.2717939

13.

Dorneich

M. C.

McGrath

K. A.

Dudley

R. F.

Morris

M. D

. (2013, October). Analysis of the characteristics of adaptive systems [Conference session]. 2013 IEEE International Conference on Systems, Man, and Cybernetics IEEE (pp. 888–893).

14.

Dudley

Dorneich

M. C.

Letsu-Dake

Rogers

Whitlow

S. D.

Dillard

Nelson

. (2014). Characterization of information automation on the flight deck. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Chicago, IL. October 27–31, 2014. 295–299 doi:10.1177/1541931214581061

15.

Ebbatson

. (2009). The loss of manual flying skills in pilots of highly automated airliners [Unpublished doctoral dissertation]. Cranfield University, Cranfield.

16.

Ebbinghaus

. (2013). Memory: A contribution to experimental psychology. Annals of Neurosciences, 20(4), 155.doi:10.5214/ans.0972.7531.200408

http://www.ncbi.nlm.nih.gov/pubmed/25206041

17.

Ericsson

K. A.

Kintsch

. (1995). Long-term working memory. Psychological Review, 102(2), 211–245.doi:10.1037/0033-295X.102.2.211

http://www.ncbi.nlm.nih.gov/pubmed/7740089

18.

Federal Aviation Administration . (2018). 91.153 VFR flight plan: Information required. Federal Aviation Administration. Retrieved April, 2018, from http://www.ecfr.gov

19.

Federal Aviation Administration . (2019). Potential flight hazards: Chapter 7-5-1: Aeronautical information manual: Official guide to basic flight information and ATC procedures. https://www.faa.gov/air_traffic/publications/atpubs/aim_html/index.html

20.

Gillen

M. W

. (2008). Degradation of piloting skills [Doctoral dissertation, University of North Dakota].

21.

Hart

S. G.

Staveland

L. E

. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.

22.

Hendrickson

S. M.

Goldsmith

T. E.

Johnson

P. J

. (2006). Retention of airline pilots’ knowledge and skill. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, San Francisco, CA. October 16-20, 2006. 1973–1976.

23.

Kornell

Vaughn

K. E

. (2016). How retrieval attempts affect learning: A review and synthesis. In Ross

B. H.

(Ed.), Psychology of learning and motivation (Vol. 65 , pp. 183–215). Academic Press.

24.

Lourda

Jose

. (2019). Effect of technology media on acquisition of cognitive skills of slow learners. In Rajakumar

Selvaraj

(Eds.), Media and child development (Vol. 2). Lulu Publication.

25.

Mengelkoch

R. F.

Adams

J. A.

Gainer

C. A

. (1971). The forgetting of instrument flying skills. Human Factors: The Journal of the Human Factors and Ergonomics Society, 13(5), 397–405.

26.

Militello

L. G.

Hutton

R. J

. (1998). Applied cognitive task analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), 1618–1641.doi:10.1080/001401398186108

http://www.ncbi.nlm.nih.gov/pubmed/9819578

27.

Minotra

Feigh

. (2017). Eliciting knowledge from helicopter pilots: Recommendations for revising the ACTA method for helicopter landing tasks. Proceedings of the human factors and ergonomics society annual meeting, Austin, TX. October 9–13, 2017. 242–246.doi:10.1177/1541931213601544

28.

Mosier

K. L.

Skitka

L. J

. (2018). Human decision makers and automated decision Aids: Made for each other? In Parasuraman

Mouloua

. (Eds.), Automation and human performance: Theory and applications. Routledge.

29.

Nakamura

. (2013). Operational use of flight path management systems. FAA.

30.

Parasuraman

Riley

. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.doi:10.1518/001872097778543886

31.

Parasuraman

Sheridan

T. B.

Wickens

C. D

. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(3), 286–297.doi:10.1109/3468.844354

32.

Prophet

W. W

. (1976). Long-term retention of flying skills: A review of the literature (No. HUMRRO-FR-ED (P)-76-35). Assistant Chief of Staff Studies and Analyses (Air Force) Washington DC Command Control And Recon Div.

33.

Rasmussen

. (1986). Information processing and human-machine interaction. North-Holland.

34.

Reys

R. E.

Bestgen

B. J

. (1981). Teaching and assessing computational estimation skills. The Elementary School Journal, 82(2), 117–127.

35.

Rose

S. R

. (1989). Members leaving groups: Theoretical and practical considerations. Small group behavior, 20(4), 524–535.

36.

Rowland

C. A

. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463.doi:10.1037/a0037559

http://www.ncbi.nlm.nih.gov/pubmed/25150680

37.

Schneider

Shiffrin

R. M

. (1977). Controlled and automatic human information processing: I. detection, search, and attention. Psychological Review, 84(1), 1–66.doi:10.1037/0033-295X.84.1.1

38.

Seamster

T. L.

Redding

R. E

. (2017). Applied cognitive task analysis in aviation. Routledge.

39.

Shiffrin

R. M.

Schneider

. (1977). Controlled and automatic human information processing: II. perceptual learning, automatic attending and a general theory. Psychological Review, 84(2), 127–190.doi:10.1037/0033-295X.84.2.127

40.

Singley

M. K.

Anderson

J. R

. (1989). The transfer of cognitive skill. Cognitive Science Series 9, Harvard University Press.

41.

Tenney

Y. J.

Rogers

W. H.

Pew

R. W

. (1998). Pilot opinions on cockpit automation issues. The International Journal of Aviation Psychology, 8(2), 103–120.doi:10.1207/s15327108ijap0802_2

42.

Tokadli

. (2015). Supporting general aviation pilots during rerouting process due to sudden weather changes [Doctoral dissertation, Georgia Institute of Technology].

43.

Volz

Yang

Dudley

Lynch

Dropps

Dorneich

M. C

. (2016, September 19–23). An evaluation of cognitive skill degradation in information automation[Conference session]. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Washington, DC.

44.

Wan

C. Y.

Huon

G. F

. (2005). Performance degradation under pressure in music: An examination of attentional processes. Psychology of Music, 33(2), 155–172.doi:10.1177/0305735605050649

45.

Werfelman

. (2015). Use it or lose it. (09/30/2020). AeroSafety World (March 2015), Flight Safety Foundation https://flightsafety.org/asw-article/use-it-or-lose-it/

46.

Wickens

C. D

. (1992). Engineering psychology and human performance (2nd ed.). Harper Collins.