Abstract
Aim. The objective of this study was to collect evidence of transfer-of-training to professional performance provided by two stand-alone PC-based flight games.
Background. These realistic games, Falcon 4.0 (F-16 specific) and Microsoft Flight Simulator (civil aircraft), are designed for entertainment purposes, lacking any purposeful or explicit instructional support.
Method. This quasi-experimental study used three pre-existing groups of gamers (n = 37; Falcon 4.0 gamers, Microsoft Flight Simulator gamers and control group: gamers without flight game experience) that performed three typical F-16 flight tasks in a high-fidelity fixed-base flight simulator.
Results. The Falcon 4.0 gamers performed substantially better on almost all tasks compared to the control group, and to a lesser degree to Microsoft Flight Simulator gamers. The Falcon 4.0 group showed near- and far-transfer on almost all flight performance measures: the game had prepared them for the generic and specific military aspects of the test flight tasks. Performance of the Microsoft Flight Simulator gamers indicated only far-transfer, i.e., transfer of more generic flight skills from the game to the test flight tasks.
Conclusion. Both near- and far-transfer of job related competences may occur by playing realistic entertainment games.
Keywords
Commercially available desktop simulations and video games are gaining interest in recent years for education and training. Desktop simulations and games are generally reality based, entertaining, interactive, rule-governed, goal-focused, and competitive (Bell, Kanar, & Kozlowski, 2008; Vogel et al., 2006). With that, they offer the opportunity to create dynamic, structured, authentic, distributed, interactive, and attractive learning environments that may offer more effective and efficient alternatives to conventional classroom education or live- or on-the-job training (Alexander, Brunyé, Sidman, & Weil, 2005; de Freitas, 2006; Korteling, Helsdingen, Sluimer, Emmerik, & van Kappé, 2011; Korteling, Helsdingen, & Theunissen, 2013; Kriz, 2010; Merchant, Goetz, Cifuentes, Keeney-Kennicutt, & Davis, 2014).
On the basis of their rapid technological advancements they are now even considered for training of tasks for which high-fidelity simulation already offered a cost-efficient solution, such as flight training. In this respect, well-known examples of desktop simulations are AirBook and LiteFlite. These video games, mostly played using a handheld device or controller, are designed primarily for entertainment purposes. Several commercial off-the-shelf games (e.g., Delta Force 2, Steel Beasts, VBS2, and Falcon 4.0) have also been adapted by the military to address military training requirements, such as tactical training (Hulst, van der Muller, Besselink, & Coetsier, 2008; Macedonia, 2002; Smith, 2010; Stehouwer, Serné, & Niekel, 2005; Zyda, 2005).
With regard to this growing interest and worldwide application of educational games and desktop simulations, it is important to establish the transfer-of-training to the real-world tasks that can be realized and the conditions under which transfer occurs. However, transfer studies are usually complex and difficult to conduct, especially for tasks or missions that are costly, dangerous or rarely conducted (Korteling, 2016; Korteling, Oprins, & Kallen, 2012; Salas, Rosen, Held, & Weissmuller, 2009), such as military (flight) operations. Therefore, this quasi-experimental study employs a relatively uncomplicated and systematic approach to collect empirical data on the training effectiveness of two commercial off-the-shelf flight games (i.e., Falcon 4.0 and Microsoft (MS) Flight Simulator) as part-task trainers for military aircraft pilot operations. This is done by comparing the performance of pre-existing experimental and control groups on operational flight tasks in a high-fidelity F-16 simulator (Figure 1).
F-16 simulator.
Transfer-of-Training
The goals of instruction and training are to improve post-training performance and to facilitate the transfer of knowledge and skills beyond particular instruction or training to support additional learning and to benefit performance in real-life environments (Korteling, van den Bosch, & van Emmerik, 1997; Roscoe & Williges, 1980). Transfer-of-training is defined as the degree to which these knowledge and skills can be applied in the real situation, or job environment (Baldwin & Ford, 1988). It is a fair generalization of the literature to conjecture that the maximum attainable amount of transfer-of-training in the first place depends on shared or common elements, i.e. physical, functional, and psychological similarities (fidelity), between the training environment and the real world (Barnett & Cici, 2002; Chase & Ericsson, 1982; Gagne, Baker, & Foster, 1949; Korteling et al., 2011; Korteling et al., 2013; Noble, 2002; Lathan, Tracey, Sebrechts, Clawson, & Higgins, 2002; Singley & Anderson, 1989;Thorndike, 1906; Thorndike & Woodworth, 1901).
It is, however, rather difficult to predict transfer-of-training outcomes on the basis of the different levels of similarity between training environment and reality. We suppose that the maximum possible amount of transfer from training to real world job is primarily determined by similarity in those task- or environmental aspects that are critical for performance of the real task and environment. Such critical aspects may concern the task goals, the task- relevant aspects of the physical environment, the (perceptual) input required to execute the task, the physical or mental actions the task performer has to execute, the behavior of other (real or simulated) characters, and the conditions under which the task has to be performed. For example, in learning to drive a car one has to learn to maintain a specific speed and distance from other objects. For such actions, direct and continuous visual input from the physical environment is critical, as well as kinesthetic input from the vehicle and the control devices such as the brake and steering wheel. However, in learning to navigate the car from point A to point B, the exact steering characteristics of the vehicle are irrelevant and other visual aspects of the world become more important, such as information from road signs, traffic lights and specific markers in the landscape (e.g. buildings, infrastructure). Thus for different (sub) tasks or different training objectives, different critical elements of the world need to be adequately represented in a simulation or game. It should be noted that we see this adequate representation as a necessary precondition for the occurrence, and maximal realizable amount, of transfer-of-training.
Next to that, transfer-of-training is affected by the (broad category of) didactical characteristics of a training system, such as, how training tasks are scheduled, what instructional interventions and support is given, and what and how feedback and debriefing are delivered (Crookall, 1995, 2010; Egenfeldt-Nielsen, 2006; Squire, 2004; Korteling et al., 2011; Korteling et al., 2013). Didactical support is supposed to help people learn, understand and remember the learning content. It aims to enhance transfer-of-training through a process of generalization and/or abstraction, from knowledge of the task at hand to a higher level of knowledge, for instance, of the general principles of a domain or general procedures for performing a class or even several classes of tasks (Van Merriënboer & Paas, 1990). Thus, the learner has the (conceptual) knowledge available to interpret what particular actions may be effective or not in what situations (Van Merriënboer, 1997). In this respect, a distinction is often made between near- and far-transfer: near-transfer tasks share structural features but differ on superficial or surface features, whereas far-transfer tasks differ from the learning tasks on both surface and structural features (Brown, 1989; Brown & Campione, 1981; Flavell, 1976; Quilici & Mayer, 1996). Near-transfer mainly depends on the degree of similarity between the job and the training and the degree to which task activities call upon the same skills. Far-transfer is more dependent upon (didactical) features of the training that enhance general or deep understanding of underlying generic task principles. All these aspects are expected to affect the efficiency and effectiveness of training and the transfer to the job that can be attained.
Finally, transfer-of-training is affected by the typical play characteristics found in games (Alexander et al., 2005; Korteling et al., 2011; Korteling et al., 2013). When games are immersive, fun, engaging, satisfying, exciting, challenging and provide players with control over scenario, they will motivate the player to continue their playful activities without the necessity to attain any external values or real-world goals. Csikszentmihalyi (1999) used the term flow to capture the state in which one continues an activity without any external goals, just for the sake of the activity. He described flow as a state of deep concentration and involvement in an activity. This state is often regarded as an enjoyable experience in which people feel active, alert, happy, strong, concentrated and creative during the experience (Seligman & Csikszentmihalyi, 2000). This flow experience is expected to enhance learning because 1) people spend more time and effort playing the game (Anette, Minogue, Holmes, & Cheng, 2009; Ericsson, Krampe, & Tesch-Römer, 1993; Williams & Kirschner, 2014) and 2) it is expected that, when a subject is in a state of flow, their brain is active showing a high degree of metabolism. This may lead to changes in structural neuronal interaction patterns and connectivity (e.g., Abbott & Nelson, 2000; Hebb, 1949).
Transfer Studies
Many quantitative reviews that have been conducted on the effectiveness of educational games (Akl et al., 2010; Bekebrede, Warmelink, & Mayer, 2011; Clark, Tanner-Smith, & Killingsworth, 2016; Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012; de Freitas, 2006; Egenfeldt-Nielsen, 2006; Girard, Ecalle, & Magnan, 2012; Hays, 2005; Ke, 2009; Lee, 1999; Leemkuil, de Jong, & Ootes, 2000; Merchant et al., 2014; O’Neil, Wainess, & Baker, 2005; Randel, Morris, Wetzel, & Whitehill, 1992; Sitzmann, 2011; Vogel et al., 2006; Wouters, van der Spek, & van Oostendorp, 2009; Young et al., 2012) have indicated that many empirical studies show limitations in their design: they do not use control groups, no pre-tests, or are confounded by differences between experimental and control groups. Furthermore, many studies only evaluate a learning effect at the end of a (game-based) training program, but do not address the transfer of acquired knowledge and skills to the real life job (Clark et al., 2016; Girard et al., 2012). One reason for this is that empirical studies into transfer-of-training are difficult to perform because it requires research at the workplace that is often difficult to organize and control (Boldovici, Bessemer, & Bolton, 2002; Cohn et al., 2009; Korteling, 2016; Korteling et al., 2012; Veldhuis & Theunissen, 2009). Measures may, for instance, be hampered by rigid training schedules, lack of control over events, logistical constraints and circumstances, limited numbers of trainees available, or lack of access to fielded systems (Cohn et al., 2009). Lack of control of all these factors may severely threaten the validity of inferences based on objective measurements of performance (Boldovici et al., 2002). Such problems also apply to our current study into if, and to what degree, some (sub) tasks of the complete suite of F-16 flight tasks can be trained using commercially available games.
The problems related to measuring F-16 pilot performance on the job are multiple: the practical problems related to unavailability of materials and F-16 pilots for longer periods of time, the fact that flight operations on an actual aircraft are dangerous and expensive, and transfer of part-task training to (whole-task) flight operations is difficult to evaluate. But a major methodological problem is the possibility that we would encounter ceiling effects in our experimental set-up. The nature of the job is such that pilots spend a lot of time training and practicing skills, up to a very high level of expertise. Thus, any training intervention might not show much effect unless it is a very big intervention in time or effort, which is usually not the case. One way to overcome this problem is to use novice pilots as participants, as was done in a study by Gopher, Well, and Bareket (1994). In this study, the researchers trained cadet pilots of the Air Force using the Space Fortress game since that game was believed to enhance flight skills and general ability of trainees to cope with the high attention load of the flight task. Flight scores of two groups of cadets who received 10 hours of training in the computer game were compared with those of a matched group of cadets without game experience. Both game groups performed significantly better than the no-game group in the subsequent test flights. They also had higher final percentage of graduation from the flight-training program (Gopher et al.).
The study by Gopher and colleagues (1994) showed that 10 hours of game play increased job performance of novice pilots when compared to novice pilots without game experience. However, in 1994 it was probably easier to find novice pilots without any game experience than it is nowadays. Finding a non-gamer control group these days will be a lot more difficult – and many of our current day (novice) pilots will have extensive experience with games that draw on similar skills as those trained by Space Fortress (Ipsos MediaCT, 2015). Thus, an intervention of 10 hours gaming using Space Fortress may not show significant flight performance benefits for our current day novice Airforce pilots.
Nevertheless, a more recent study was able to find non-gaming comparison and control groups. A study by McKinley, McIntire, and Funke (2011) evaluated whether playing video games benefits Unmanned Aerial Vehicle (UAV) pilot operations. They found that skills learned in video game play may transfer to novel environments and improve on the job performance. They used pre-existing groups of 1) video gamers, 2) non-gaming UAV pilots and a 3) control group who were neither video gamers nor pilots, and compared their performance on a set of cognitive tasks that represented UAV pilot tasks. The researchers found that, although the UAV pilots outperformed the gamers and control groups on multi-attribute cognitive tasks, the gamers outperformed pilots on cognitive tests related to visually acquiring, identifying, and tracking targets. In addition, both the gamers and pilots performed similarly on the UAV landing task, but outperformed the control group (McKinley et al.).
The study of McKinley and colleagues (2011) is especially interesting because of their use of pre-existing groups and criterion tasks that were representative of - but not the real- UAV pilot tasks. Thus, the gaming intervention can be substantial compared to other training efforts by setting a minimum amount of game-experience (in hours or years) as a selection criterion to participate in the experimental group. Secondly, because the criterion performance was measured not on the job, but using tasks that represented the job adequately, more participants could be recruited for the experiment and availability of operational equipment was no issue. As such, the study presents a relatively simple and systematic approach to generate empirical data on the training value of games.
The abovementioned studies show that video gaming – whether that is a specific game such as Space Fortress, or generic game experience as the participants in the McKinley and colleagues (2011) study – may enhance skills that are necessary for controlling an airplane. In the Gopher and colleagues (1994) study, the game was embedded in two different training methods, one aimed to train specific control skills, the other targeted at acquiring a more general ability to cope with high task demands of a typical pilot job. Gopher and colleagues found benefits of both training methods, which can be explained by the ideas of Van Merriënboer (1997) that specific didactical aspects of a game or game-based training course can enhance deep understanding or general procedures for performing a class or even several classes of tasks. In the McKinley and colleagues study, unspecified and varied game experience enhanced later performance on UAV specific tasks. This can be the result of the practice variability (Burke & Hutchins, 2007), or the didactical aspects of specific games the participants played (Wulf & Shea, 2002), or similarities between the game and the criterion tasks, although the latter is unlikely, given the fact that the researchers did not control the game experience of participants.
The results of both game transfer studies discussed above warrant investigation into whether it is the specific game that enhances the job performance, or (flight) gaming experience in general. Therefore, in the present study we compare performance on F-16 flight tasks of a group of gamers who play an F-16 specific flight game (Falcon 4.0) with the performance of gamers who play civil aircraft flight game (MS Flight Simulator) and with performance of gamers who have no flight game experience (control group).
Falcon 4.0 and MS Flight Simulator
Both MS Flight Simulator and Falcon 4.0 are games typically intended for players who dream of becoming an airplane pilot. MS Flight Simulator is one of the longest-running, best-known and most comprehensive home flight simulator series, representing civil aircraft. It is a computer program for Microsoft Windows simulating many kinds of aircraft. Throughout the years the package has been developed as a very detailed and realistic simulation of actual flight including almost all available avionic systems. The flying area encompasses the whole world, to varying levels of detail, including numerous civil airports. Detailed scenery can be found representing major landmarks and an ever-growing number of towns and cities. The US navy provides a customized version of Microsoft Flight Simulator to student pilots and has installed it in several Naval Air Stations (Macedonia, 2002).
Falcon 4.0 1 is a realistic military air combat simulation of the F-16 Fighting Falcon jet fighter in a full-scale modern war set. Falcon 4.0 game play parallels actual fighter pilot combat operations (Lenoir & Lowood, 2002; Prensky, 2001). First, over 30 training scenarios acquaint the player with F-16 manoeuvring, avionics operation and various United States Air Force protocols. Falcon 4.0 playing starts with practice, for which a huge variety of training scenarios is available. These scenarios include common situations, such as: landing during an engine flameout, navigation using on-board instruments, avoiding threats and deploying various weapons against air and ground targets. After training, the player may start the primary game play mode in the campaign, that simulates participation in a modern war. Alternatively, the player may engage in dogfight mode providing an individual air engagement, or create what are effectively miniature campaigns, known as Tactical Engagements. The results of the players’ performance while using Falcon 4.0 are used to generate a logbook. This contains such details as flight hours, Air-to-Air and Air-to-Ground kills, decorations, a name and photo and the current rank of the player. Good performance (such as eliminating large numbers of enemy ground units or surviving a difficult engagement) during a mission may lead to the award of a decoration or promotion, while poor performance (destroying friendly targets or ejecting from the aircraft for no good reason) can lead to court-martial and demotion.
The Present Study
The current study follows a similar approach as the McKinley and colleagues (2011) study. In our quasi-experimental research we collected empirical data on the transfer of training of (flight) games using pre-existing experimental and control groups (Falcon 4.0 gamers vs. MS Flight Simulator gamers vs. non-flight gamers) and evaluate their flight operation performance on a high-fidelity flight simulator (Figure 2 and 3) representing operational F-16 flight tasks. Apart from the aforementioned benefits of this approach for availability of experimental groups with ample (flight) game experience, control groups, and equipment we also have better experimental control over test scenarios in a simulator than on the actual aircraft, and it provides an opportunity to only assess specific (part-task) skills by creation of specific scenarios and automated support.
F-16 simulator: Exterior.
F-16 simulator: Dome.
We use three different test tasks: Basic Flight, Tactical Formation and Close Formation. These tasks differ in their military character: from basic flight to tactical and close formation the flight tasks involve more specific military aspects like sharp angles, and flying at close distance from partner aircraft. As we argued in the previous section, the Falcon 4.0 game includes general and specific elements of military flight in a playing environment that represents many of the critical aspects of the real world (F-16) flight tasks. Therefore, we may expect relevant flight skills to transfer to the F-16 simulator tasks and we expect Falcon 4.0 gamers to perform better on these real world tasks than MS Flight Simulator gamers who have been practicing flight in a generic (civil) environment that does not represent the critical aspects of military flight tasks. Both the Falcon 4.0 and MS Flight Simulator games do not include specific didactic facilities and instructional support to facilitate learning, although informal coaching and feedback among community members is not unusual. The expected near-transfer is therefore primarily determined by the similarity of the games with the real-world task and by the level of practice the gamers have experienced. Some far-transfer of skills are expected from the MS Flight Simulator game to the F-16 simulator, specifically for the Basic Flight task, being the most generic of the three test tasks.
Concluding, we hypothesized that:
Participants from the Falcon 4.0 group will show better performance on the simulator test tasks than participants from the MS Flight Simulator group and the control group.
The MS Flight Simulator gamers will perform better at the simulator test tasks than the control group.
Performance differences between these groups will be larger in the tasks that are very specific for F-16 flight (i.e. near-transfer) and smaller in tasks that invoke more generic flight elements and procedures (far-transfer).
Method
Participants
The study applied a quasi-experimental approach with two pre-existing experimental and one control group recruited by convenience sampling groups of gamers: Twelve were recruited from the online Falcon 4.0 community, twelve were recruited from the online MS Flight Simulator community, and thirteen were gamers recruited from the experiment participants database from Netherlands Organization of Applied Research (TNO). This latter group did not have any experience with flight (simulation) games (control group). Participants were paid for their participation. All participants reported to be healthy and had normal or corrected-to-normal visual acuity. The Falcon 4.0 gamers were between 34 and 62 years old (M = 44.9, SD = 9.2) and their experience with the flight game ranged from 2 to 20 years (M = 8.9, SD = 4.3), playing between 0 and 20 hours a week (M = 7.4, SD = 7.2). The 0 hours per week was a Falcon 4.0 gamer with 10 years of game experience, but currently not actively gaming. Three of the Falcon 4.0 gamers had actual flying experience, ranging from 80 to 7000 flight hours, but not on an F-16.
The MS Flight Simulator gamers were between 43 and 78 years old (M = 56.8, SD = 9.8), with 1 to 30 years of Flight Simulator experience (M = 12.3, SD = 7.5) playing between 0 and 15 hours per week (M = 4.5, SD = 4.9). In this group, three experienced Flight Simulator gamers (between 7 and 10 years of experience) had not played the game recently, therefore put 0 hours per week. Furthermore, two men had actual flight experience: 41 and 485 flight hours respectively, again not on an F-16.
The control group participants were between 22 and 50 years old, (M = 34, SD = 8.5). One of these participants had actual flight experience, not on an F-16, of 200 flight hours.
Materials
Pre-test questionnaire: A questionnaire collecting personal data from participants (e.g., age, job experience, education, flight gaming experience) was administered before the test started.
Pre-test task: To determine participants’ visuo-motor performance, the Digit Symbol Substitution Test of the Wechsler Adult Intelligence Scale (Wechsler, 1981; WAIS-IV-NL, see pearsonclinical.nl) was used.
Task environment: The task environment was a high-fidelity F-16 Aircraft simulator that is developed by the Netherlands Organization of Applied Research (TNO) for research purposes. During the development this simulator has been evaluated by 20 F 16 pilots of the Dutch Royal Airforce. They rated the tactical protocols and multi-functional display as having a real look and feel.
This simulator consists of a cockpit mock-up of the F-16 M3 aircraft. In this mock up, the controls were hands on throttle and stick (HOTAS; Aerotronics Ltd. Florida – US). The mock-up was placed in front of a dome screen (Barco SHEER 8) that provided a 240° field of view (FOV) horizontally and 155° FOV vertically (Figure 2 and 3). The dynamic Flight model was a black box F-16 model provided by the producer of the F-16 platform (Lockheed Martin). The head up display (HUD) was projected via a separated beamer on the inside of the dome and was visible through the HUD-visor of the mock-up. The HUD displayed the following information:
The flight path marker, indicating the flight direction of the aircraft.
Two ladders, one on each side:
Speed (knots) Altitude (feet)
Heading - the direction that the aircraft’s nose is pointing
A Multi-Functional Display was placed in the mock-up, on right hand side of the participant, controlled by a Dell personal computer (PC). On this display the aircraft (green object in middle of display) and flight path (the white box) were shown in God’s view mode.
The projected landscape represented the area around the city of Nelis airbase, Las Vegas in the State of Nevada of the United States of America. This area database was generated by an Evans & Sutherland/Rockwell Collins EPX-50 system. 2 F-16 aircraft behaviour was according to an F-16 flight model generated by a Simigon simulation engine. 3
Test tasks: On the basis of several sessions in which experienced F16 test-pilot of the Royal Dutch Air Force flew in the simulator comparing its’ functions and behaviour relative to real F16 flight, three simulator flight tasks were selected. Although the simulator lacks capacity for vestibular stimulation of motion and acceleration, these tasks were considered being very representative of the complexities of actual F-16 flight. These tasks were used as criterion tasks for our experiment, i.e.: (1) Basic Flight, (2) Tactical Formation as a wingman and (3) Close Formation as a wingman. Basic flight is mainly a generic flight task – albeit using the military aircraft controls – with some minor military specific details. Tactical and close formation also include generic task elements of flight, but in addition are characterized by some very specific F-16 task elements. These will be discussed in more detail below.
Basic flight: In this task, participants had to navigate the aircraft over 9 pre-determined waypoints (Figure 4). Each waypoint had to be reached with a specific altitude and speed. The flight legs from waypoint to waypoint were flown with heading north, east, south, and west. Waypoints were placed 10 miles apart. Participants started their run 10 miles south of the first waypoint at an altitude of 5000 feet and with heading 000 (North). The experimenter instructed participants to make the sharp right turn after each waypoint, and communicated the required speed and altitude that had to be reached as fast as possible. Participants also had a kneepad presenting information on the nine waypoints. Flying over waypoints while maintaining adequate heading, altitude- and speed is a common and generic flight task, seen in all kinds of civil and military flight. Specific F-16 characteristics of this task were the requirement to make turns with substantial banking angles using the F-16 interfaces of the simulator. This way of making sharp turns over waypoints is less common in non-military flight situations.
Waypoints with the required speed and altitude in Basic Flight.
Tactical formation as a wingman: In this task participants were instructed to fly as a wingman in a two-ship formation (Figure 5). The lead aircraft was controlled by a software agent and flew a predefined route. Participants were instructed to follow the lead in all turns. Participants started their task flying at a so called line abreast position: at a rather long distance of one mile right off the lead at an altitude of 11000 feet and airspeed of 420 knots. The lead performed 90 degree or 180 degree turns (60 degree bank-angle), which implied a wing change at every turn to stay in formation (for example see Figure 6). When participants were flying 30 seconds in the line abreast position the experimenter commanded “Two, 90 left/right, go” or “Two, in place left/right, go”, in response to which the participants had to make the 90 degree turn or 180 degree turn respectively. The experimenter had the participants switch wings by the “Two, switch left” command to insure that the participants would fly all combinations of turns possible, i.e. 2 (left or right turns) * 2 (left or right position relative to the lead) * 2 (90 or 180 degree turn) = 8 turns. The formation flying with wing changes is a task typical for military flight, not for civil flight.
Tactical formation as a wingman.
Tactical formation as a wingman: 90 degree turn left.
Close Formation as a wingman: In this task participants were instructed to fly as a wingman in a two ship close-formation (Figure 7). The lead aircraft was controlled by a software agent who flew a predefined route with constant airspeed and altitude. Participants were instructed to follow at a fixed distance and position relative to the leading aircraft, they could reach and monitor the correct distance and position by visually aligning the missile tip and gun pot of the leading aircraft. Participants started their run one mile behind the lead at an altitude of 11000 feet and airspeed of 420 knots. When participants had reached the correct distance and position the experimenter would command the lead to make a turn. A total of eight right turns and eight left turns were performed each at four different bank-angles (15, 30, 45, or 60 degrees) and with the wingman at the left (eight turns) or right wing (eight turns) of the lead. In between turns, the experimenter could instruct participants to make a wing change. Again, the formation flight, this time following the lead aircraft at a very close distance and the very sharp, bank angles make this a typical military flight task. Pilots are required to apply tactical procedures and have maximal awareness of one’s own aircraft relative to the lead aircraft.
Close formation as a wingman.
Procedure
The experiment was conducted in individual sessions of four hours per participant. On entering the room, participants were first given a briefing on the purpose of the experiment. Subsequently, they had to fill in a questionnaire on personal data such as their age, educational level, and experience with playing PC flight simulator games on a PC. Then they performed the WIAS Digit Symbol substitution task in order to determine their basic visuo-motor ability (capacity), which may be considered relevant for flying an airplane. Next, they were given instruction on the test tasks they had to perform after which they received 15 minutes of practice time in the simulator. During this practice time, participants could practice the Basic Flight, but using different coordinates than in the actual experimental task. Participants could practice F-16 flight in the simulator using throttle, stick, HUD and Multi-Functional Display. After this practice, participants executed the three test tasks in order of increasing similarity with F16 flight: first basic, then tactical and last close formation. Before each consecutive test task, they evaluated the preceding test task on its complexity and their own performance, after which they had a short break of 15 minutes and received instructions for the next test task. After the experimental sessions, the participants were extensively debriefed.
Performance Evaluation
In consultation with an F16 test-pilot of the Royal Dutch Air Force, different performance variables were defined for the three tasks and data were collected that represented the quality of real flight performance. For Basic Flight, the difference between required speed and altitude and actual speed and altitude at each of the 9 waypoints was calculated. Small difference scores meant better performance. For Tactical Formation, participants’ performance was measured by evaluators who were not informed about the background of the experiment and participants: The participants moment of turning was judged to be correct (0) or incorrect (1) in response to the lead aircraft instructions or actions, and coming out of every turn, the heading and speed in relation to the lead aircraft was judged to be correct (0) or incorrect (1). Thus, per turn, 3 evaluations were made, and the number of incorrect actions were tallied and analysed separately. For the Close Formation task, the blind evaluators tallied the number of outs, busts and resets. An out meant that the participant did not keep his aircraft in the same plane, a bust was a situation in which the participant had lost visual sight of the lead aircraft and a reset meant that the experimenter had to restart the session to get the participant and his lead aircraft back in formation because the participant was not able to steer his aircraft back in formation within two minutes after losing his lead. The number of outs, busts and resets had a high internal consistency (Cronbach α = .97), thus these were aggregated and analysed as one performance score.
Results
Data on participants’ age, Flight Simulation game experience, visuo-motor performance, and actual Flight Experience is presented in Table 1. For all the test tasks, the mean performance scores per task are presented in Table 2.
Means and Standard Deviations of Age, Digital Symbol Substitution Task (WAIS) Scores, Actual Flight Hours and Flight Simulation Game Experience for the Falcon 4.0 Gamers, the MS Flight Simulator Gamers and the Control Group.
Means and Standard Deviations of the Performance Scores on Basic Flight, Close Formation and Tactical Formation for the Falcon 4.0 Gamers, the MS Flight Simulator Gamers and the Control Group. Higher scores is worse performance on all measures.
In the analyses reported below, a significance level of .05 was set, and partial eta-squared or Adjusted Hedges are reported as a measure of effect size. Missing data were replaced by the mean of 4 nearby data points, a standard method in SPSS 18.0. 4
The pre-existing groups were analysed with respect to factors that may affect test task performance: Their flight simulation gaming-experience in years and hours per week, their age, the amount of actual flight hours, and visuo-motor ability as measured by the Digit Symbol Substitution Test of the Wechsler Adult Intelligence Scale (WAIS). First, a Multivariate Analysis of Variance with Game Type (MS Flight Simulator or Falcon 4.0) as between-subjects factor and hours per week and years of experience as dependent variables was conducted, using only two of the three groups of participants because the control group did not have any flight simulation experience. The results showed that Falcon 4.0 participants and MS Flight Simulator participants did not differ significantly in their experience with playing flight simulation games (Wilks’s Lambda = 0.87, F (2, 21) = 1.47, ns). Secondly, a Multivariate Analysis of Variance was conducted with Game Type (Falcon 4.0, MS Flight Simulator and Control) as between-subjects factor and Age, amount of Actual Flight Hours, and WAIS score as dependent variables. The results showed that the Game Type groups differed on these scores, Wilks’s Lambda = .409, F (2, 64) = 6.01, p <.01, ηp2= .36. Further analyses showed that Age differed between the groups (F (2, 34) = 19.23, p < .01, η p 2 = .53, MSE = 83.98) as well as WAIS scores (F (2, 34) = 5.46, p = .01, η p 2 = .24, MSE = 100.89) but amount of Actual Flight Hours was not significantly different among the groups (F (2, 34) = 1.57, ns). Pairwise comparisons revealed that control group (M = 34.00, SD = 8.53) were younger (p = .01) than Falcon 4.0 participants (M = 44.92, SD = 9.16), and both these groups were younger (p < .01 and p = .01 respectively) than MS Flight Simulator participants (M = 56.75, SD = 9.81). With respect to the WAIS scores, pairwise comparisons showed that MS Flight Simulator participants (M = 50.8, SD = 11.6) scored significantly (p = .01) lower than control group (M = 63.8, SD = 8.4), but scores of MS Flight Simulator and Falcon 4.0 (M = 59.8, SD = 10.0), as well as from Falcon 4.0 participants and the control group did not differ. Based on these results, we decided to incorporate Age and WAIS scores as covariates in our analyses of the task performance of the different groups.
Basic Flight
During the Basic Flight, at each waypoint the absolute deviation between required and actual altitude and speed were calculated. Altitude deviation scores over all groups ranged from 0 to 3980 feet (M = 109.1, SD = 134.5), Speed deviation scores ranged from 0 to 189 knots (M = 109.1, SD = 134.5). Mean deviation scores for altitude and speed were calculated for each participant. To analyse the effects of type of gaming experience on participants’ altitude and speed deviation scores (Table 1), a Multivariate Analysis of Covariance was conducted, with Flight Game Type (Falcon 4.0, MS Flight Simulator or control) as between-subjects factor and Age (years) and WAIS score as covariates. Substantial main effects of Flight Game Type, Wilks’s Lambda = .56, F (2, 62) = 5.15, p < .01, η p 2=.25, and Age, Wilks’s Lambda = .83, F (2, 31) = 3.21, p = .05, η p 2=.17 were found (see Figure 8a, b). More detailed analyses shows that Flight Game Type had both an effect on the Speed and Altitude scores: F (2, 32) = 10.7, p < .01, η p 2=.40 and F (2, 32) = 4.49, p = .02, η p 2=.22, respectively. Subsequent pairwise comparisons show that the Falcon 4.0 group (Malt = 28.8, SDalt = 26.6, Mspeed = 4.5, SDspeed = 3.7) outperformed the control group (Malt = 115.6, SDalt = 66.1, Mspeed = 19.6, SDspeed =13.0) on Altitude (p = .03) and on Speed (p < .01). Furthermore, the MS Flight Simulator participants (Malt =182.3, SDalt = 202.1, Mspeed = 16.7, SDspeed = 13.60) also outperformed the control group, but only on the Speed scores (p = .04).
Basic Flight performance.
Basic Flight performance.
Tactical Formation
For each of the 8 turns, the moment of turning was judged to be correct (0) or incorrect (1), and coming out of every turn, the heading and speed in relation to the lead aircraft was judged to be correct (0) or incorrect (1). Thus, per turn, 3 evaluations were made, and the number of incorrect actions were tallied and analysed separately (see Table 2). Over all groups, the mean score (Mturn) for moment of turning was 1.7 incorrect actions (Min = 0 Max = 8, SD = 1.8), mean score (Mhead) for Heading was 1.3 incorrect actions (Min = 0, Max = 6, SD = 1.7) and for Speed Mspeed = 4.00 incorrect actions (Min = 0, Max = 8, SD = 3.2). Multivariate Analysis of Covariance was conducted with Game Type (Falcon 4.0, MS Flight Simulator and control) as between-subjects factor, Age and WAIS scores as covariates, and the amount of incorrect actions at Moment of turning, Heading and Speed as dependent variables. As shown in Figure 9a, b and c, Game Type had an effect on the Tactical Formation scores (Wilks’s Lambda = .568, F (6, 60) = 3.27, p < .01, η p 2 = .20). Further analyses show that the Game Type groups scored differently on both Moment of turning (F (2, 32) = 3.97, p = .03, η p 2 = .20, MSE = 2.60) and Speed (F (2, 32) = 4.81, p = .02, η p 2 = .23, MSE = 8.49), but not on Heading (F (2, 32) = 2.04, ns). Pairwise comparisons show that Falcon 4.0 participants (Mspeed = 2.50, SDspeed = 3.45; Mturn = 0.75, SDturn = 0.86) were better than the control group (Mspeed = 6.23, SDspeed = 1.74; Mturn = 1.92, SDturn = 1.93), both on the Speed (p = .01) and the Moment of turning (p = .04). The MS Flight Simulator participants did not score significantly different from either the Falcon 4.0 or the control group.
Tactical Formation performance.
Tactical Formation performance.
Tactical Formation performance.
Close Formation
The number of outs, busts and resets had a high internal consistency (Cronbach α = .97) and were thus aggregated into one Close Formation score. The minimum score over all groups was 1, the maximum score was 155 (M = 52.7 SD = 50.3). A Univariate analysis of Covariance was conducted with Game Type (Falcon 4.0, MS Flight Simulator and control) as between-subjects variable; Age and WAIS score as covariates and Close Formation score as dependent variable. Results showed that Game Type had a significant effect on the Close Formation score, F (2, 32) = 8.77, p < .01, η p 2 = .35, MSE = 1817.85, see Figure 10. Pairwise comparisons showed that MS Flight Simulator (M = 78.3, SD = 56.5) and the control group (M = 68.2, SD = 43.1) did not differ on the Close Formation scores, but Falcon 4.0 participants (M = 10.2, SD = 8.5) were better than both MS Flight Simulator (p < .01) and control group (p = .01).
Close Formation performance.
Discussion
In the present experiment we investigated transfer-of-training of two flight simulation games that have not been designed for training, to F 16 flight tasks. In a quasi-experimental design (Campbell & Stanley, 1963), F-16 flight performance of pre-existing groups of Falcon 4.0 and MS Flight Simulator gamers was compared mutually and to a control group of gamers who had no experience with flight games. The Falcon 4.0 game includes generic elements of (civil) flight as well as many specific military elements of F-16 flight. The MS Flight Simulator game doesn’t include specific military flight elements but the flying of civil aircraft in this game was expected to provide players with generic task aspects of the flight domain, such as flying over waypoints and maintaining a certain altitude.
In the introduction, we formulated a basic precondition, that educational games should always have a certain degree of similarity, or fidelity, with respect to the relevant real-world task. That is, the maximal obtainable transfer is in the first place determined by the amount of shared or overlapping elements (physical, functional, psychological) between the training environment and the real world (Gagne et al., 1949; Korteling et al., 2013; Lathan et al., 2002; Noble, 2002; Thorndike & Woodworth, 1901). Therefore, we selected three criterion flight tasks with increasing degrees of specific military F-16 flight aspects. We explicated that only the Falcon 4 game includes many of these F-16 specific task elements (Lenoir & Lowood, 2002), whereas more generic task elements are present in both flight games. On the basis of these differences between the games, we expected that Falcon 4.0 gamers would show a high degree of transfer to F-16 simulator test tasks (far- and near-transfer) and that the MS Flight Simulator group would show a more limited amount of transfer from game to the F-16 test tasks (only far-transfer). Both expectations were confirmed by the overall pattern of the results. In Basic Flight both the Falcon 4.0 and the MS Flight Simulator group outperformed the control group, but performance of these two flight gamer groups did not differ significantly from each other. In the Tactical Formation task, only the Falcon 4.0 gamers outperformed the control group. Here, performance of the MS Flight Simulator group was in between the Falcon 4.0 and control group, not differing significantly from either. In Close Formation the Falcon 4.0 group performed substantially better than both other groups, and no difference occurred between the MS Flight Simulator group and the control group.
In general, the differences in task performance between the three groups of participants reflect the degree to which their game included task elements of civil or F-16 flight. That is: Falcon 4.0 gamers, who have been trained with specific military flight task elements, showed superior performance on most tasks. Nevertheless, MS Flight Simulator participants’ performance neared performance of Falcon 4.0 gamers in Basic Flight and Tactical Formation, tasks that contain less specific military flight aspects. We suppose that the MS Flight Simulator group had a certain level of understanding and flight skill enabling them to choose effective courses of actions in some flight tasks. This counts particularly in more generic elements of flight tasks in which performance is more dependent on general knowledge of the flight domain or more general flight procedures (Van Merriënboer & Paas, 1990). This transfer of overall flight skills may be considered as far-transfer (Brown, 1989; Brown & Campione, 1981; Flavell, 1976). In contrast, Close Formation is a task requiring specific F-16 skills including complex tracking dynamics, following of the lead aircraft very closely at the same altitude and with the same banking angle. This is typically seen in F-16 flight (and represented in Falcon 4.0), but rather unusual in civil flight (MS Flight Simulator). Here, generic flight skills or meta-knowledge of flight tasks may not be sufficient: very specific flight skills are required that can only be practiced under circumstances closely resembling the actual F-16 environment and constitutes near-transfer.
The use of games in learning and instruction, often referred to as educational or instructional games, has been propagated by many researchers. Hays (2005) has reviewed 48 empirical research articles on the effectiveness of instructional games. This extensive report also includes summaries of 26 other review articles and 31 theoretical articles on instructional gaming. For our study, two major conclusions of the report seem particularly relevant: the first one being that instructional games are more effective if they are embedded in adequate instructional programs that include debriefing and feedback and the second one that instructional support during play increases the effectiveness of instructional games. Based on this, Hays recommends that serious games should preferably be designed to meet specific instructional objectives and be used accordingly as adjuncts and aids, not as stand-alone instruction. In fact, several (meta-) reviews on instructional effectiveness of simulation, virtual reality and games, draw similar conclusions concerning the critical role of the instructional context (Clark et al., 2016; Merchant et al., 2014; Sitzmann, 2011; Wouters et al., 2009). This specific instructional context entails something different for various types of studies. Sometimes the learning gain of additional instruction outside the game context is studied, and mixed results are found. Clark and colleagues find no benefit from additional instruction, whereas both the results of Wouters and colleagues (2009) and of Sitzmann (2011) indicate it does improve learning. On the role of teachers or instructors, evidence is also ambiguous: Clark and colleagues found that scaffolding by a teacher enhances learning from games when compared with system-based scaffolding, but Merchant and colleagues conclude that the availability of a teacher does not make a difference in the learning gain from games.
Our showed that stand-alone games without support by an instructor, coach or automated intelligent tutoring system may provide substantial transfer to real-world tasks. Of course this does not mean that the recommendation to include instructional support with good instructions, feedback, and debriefing is trivial or untrue. Our study showed that under certain circumstances, dedicated instructional support is not always necessary for positive transfer of training. This is also confirmed by Petty and Barbosa (2016) who provide empirical evidence that self-guided study and simulation-based practice without an instructor can produce improvement in complex psychomotor skills. In line with other researchers (e.g., Gagne et al., 1949; Lathan et al., 2002; Noble, 2002; Thorndike & Woodworth, 1901), we suppose that the occurrence and the maximum attainable amount of transfer-of-training in the first place depends on shared or overlapping content (i.e. elementary physical, functional, and psychological similarities, between the training environment and the real world). It is thus still valid to expect that transfer, given this fidelity prerequisite, will be (substantially) enhanced by good instructional features of the game (Hays, 2005; Sitzmann, 2011). In fact, games such as Falcon 4.0 and Microsoft Flight Simulator include video instructions, provide ample opportunities for playback, they have an active community in which peer-feedback and peer-tutoring are not uncommon. Although not intended, these features may serve as instructional or didactical support and have probably contributed to the transfer-of-training that we found.
Apart from the fidelity and any beneficial effects of didactical game features, the positive transfer of skills from the games to the real world F-16 operations may have been fostered by attractive play features that enhance active engagement and frequent experience with the game. These game play or entertainment features discriminate educational gaming from conventional educational simulation (Korteling et al., 2013). Although the added value of entertainment is debated in literature, with Sitzmann (2011) concluding entertainment does not serve learning directly, the active engagement during learning – whether that is in game based learning or alternative instruction – does enhance its effectivity. Furthermore, the entertainment and engagement aspect of games may have people play for longer periods of time. And from previous studies it does become clear that trainees learn more, relative to comparison groups, when they play longer periods and can access the game as many times as desired (Kirschner & Williams, 2014; Sharek & Wiebe, 2014; Sitzmann, 2011). This points to a possible limitation of the present study: the amount of time that trainees spent gaming was not controlled and therefore we cannot say much about the efficiency of the game based training, only about its effectiveness (amount of transfer of training). Nevertheless, although we have not established the efficiency of the game-based training using Falcon 4.0 or MS Flight Simulator, we assume that the games offer a cost-effective way of (basic) training of novices, because people tend to play during their private time at home, bringing savings on instructional infrastructure and personnel. The present flight games include several strong game-play features such as: goal-directedness, competition, having a framework of agreed rules (Lindley, 2004; Whitton, 2011), and providing feedback to enable players to monitor their progress towards the goal (Prensky, 2001; Wouters et al., 2009). These features must have motivated people to play with a high degree of engagement. A next step in the evaluation of the Falcon 4.0 game for F-16 flight operations could be a true transfer-of-training experiment in which the amount of game -training needed to reach a certain level of skill is compared to alternative training methods.
Footnotes
Acknowledgements
The authors like to thank Eric Cornelissen, Jan Hilt, Wytze Ledegang, Erwin Neyt, and Martin van Schaik for their support in executing the experiment.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research has been supported by and the GATE project (funded by the Netherlands Organization for Scientific Research and the Netherlands ICT Research and Innovation Authority) and by the Dutch Ministry of Defense.
Notes
Author Biographies
Contact:
Contact:
Contact: