Success factors of evidence-based training for pilots

Abstract

Training pilots is a long-lasting and important task for the aviation safety and proper operation of airlines. Evidence-Based Training (EBT) is proposed as one of the best alternatives for pilot training. This study identified the need of EBT in terms of non-technical performance training for pilots. The purpose of this paper is to identify the need to introduce a new training system, the EBT model, and to assess and verify its performance by operating airlines to assist in decision making by airlines considering the introduction of EBT. A total of 1003 people were subjected to the study, including 557 captains and 446 co-pilots of each model who responded to the survey conducted after EBT. Results of the survey were analyzed through multiple regression analysis to confirm the validity of this model, and the pilots who were trained were also satisfied with EBT. The results show that the training program information and the system quality have a positive effect on the satisfaction of the trainee and the enhancement of the trainee's perceived ability. The results of this research contribute a positive basis for EBT. The implication of this study is that the proper provision of training program and the better system could support the performance of training for pilots.

Keywords

pilot training evidence-based training IS success model

Introduction

As one of the proper training practices, evidence-based training (EBT) has been suggested and spotlighted in terms of aviation safety (ECA, 2019; IATA, 2013b; ICAO, 2013; Moro, 2019). Initiatives such as crew resource management, safety management systems and the automated cockpit have played a part in increasing safety levels, but the latter can also pose challenges if training is inadequate (Allianz Global Corporate & Specialty, 2014). In 2013 the International Air Transport Association (IATA) presented a new training method jointly with the International Civil Aviation Organization (ICAO) and the International Federation of Air Line Pilots’ Associations (IFALPA).

The airline industry continues to grow at record annual rates, with a steady increase in the number of travelers using air transport, although facing a temporary decline in the COVID-19 situation. This development of the aviation industry would not have been achieved without the assurance of aviation stability. Due to the nature of aviation accidents that lead to major accidents, the airline industry's interest and investment in aviation safety are huge and will inevitably increase.

In the past, successive airline accident investigation bodies have learned that many aviation accidents are due to human factors, and that the Crew Resource Management (CRM) and Line Oriented Flight Training (LOT) were introduced as training methods to complement pilots’ non-technical performance. Nevertheless, the Advanced Qualification Program (AQP) was developed and defined in the U.S. to reduce the airline accident rate, and the European version of the Advanced Training and Qualification Programme (ATQP) was adopted in Europe. IATA has developed a more advanced training program, EBT, with ICAO and IFALPA. This new training paradigm is now spreading all over the world.

All new training systems experience resistance from stakeholders, such as airlines and pilots, during their introduction. The same is true of EBT. Airlines, for financial reasons, and pilots often oppose the introduction, citing the importance of technical skills training. Resistance to new things is a natural phenomenon. Despite the resistance, the key is to increase the satisfaction level of pilots, who are the consumers of the new training program, to adopt and get used to EBT. Training hours and expenditure lead to increased sales, value added, employment, productivity, and exports (Martins, 2021).

This study first looked at the need to introduce EBT through a literature review of the history of pilot training methods development. In addition, the information system success model of DeLone and McLean (1992) was used to measure the performance of EBT introduction, and empirical analysis was performed on the effectiveness of EBT (DeLone and McLean, 1992).

Evidence-based training

Evidence-based training (EBT) is a general terminology in many industries, especially in the professional sectors, such as medical, clinic, and aviation. In healthcare sector, some research show the performance of EBT (Morshed et al., 2017; Morton et al., 2019; Rosen et al., 2017; Wilfley et al., 2018), while in the aviation industry there are few academic literatures on the performance of EBT. Evidence-based training and procedures is useful to increase the safety of first responders (Hembroff et al., 2018). Objective, real-time data can be used for performance appraisal, training and development, logistical tracking, wellness programs, and employee safety (Tomczak et al., 2018).

According to an IATA Annual Review, the number of air passengers worldwide increased from 106 million in 1960 to 4.1 billion in 2017, a number half of the world's population, and 41.8 million flights were safely operated in that year (IATA, 2018). This ever-increasing amount of air traffic has been an opportunity for growth and a challenge for aviation safety. A safety report from ICAO said 2017 marked 12.2 deaths per million passengers worldwide, making it the safest year since aviation records began (ICAO, 2018). With the development of manufacturing technology, engine reliability has been greatly increased to the point where there is little chance of failure in flight (Allianz Global Corporate & Specialty, 2014).

As new accidents occurred, training requirements became larger and the pilot's regular training became inefficient with only a list to be completed (ICAO, 2013). Table 1 shows a short history of evidence-based training.

Table 1.

Short history of evidence-based training.

Year	Description
1979	NASA and the aviation industry host Cockpit Resource Management workshop for Crew Resource Management (CRM)
1981	United Airlines introduced CRM training – Crew Leadership and Resource Management (CLRM)
1981	NASA and the airline industry's workshop on the line-oriented flight training (LOFT) guideline
1992	FAA developed the Advanced Qualification Program (AQP). AQP is a training method that requires voluntary participation to meet the training, evaluation, qualification and maintenance of pilots. AQP covers CRM and Line Operational Simulations (LOS).
2006	ATQP (Alternative Training and Qualification Program) was adopted and defined.
2013	ICAO developed evidence-based training (EBT), a new methodology for the development and implementation of regular training and evaluation and issued a revision of Doc 9868 “PANS-TRG” and Doc 9995 “Manual of Evidence-based Training”.

Source: Based on (CAA, 2016; Helmreich et al., 1999; IATA, 2013b; ICAO, 2013, 2015; Lauber and Foushee, 1981; Weitzel, 1992).

Most operators currently conduct CRM training annually, with its benefits widely recognized. A focus on the human factors could greatly improve flight crew awareness on a regular basis (Kelly and Efthymiou, 2019). CRM and EBT have a complimentary relationship to each other. EBT enforces instructors to analyze the root causes to correct inappropriate actions, rather than simply asking a flight crew member to repeat a maneuver with no real understanding as to why it was not successfully flown in the first instance. EBT does not only bring safety benefits to those who implement it, it also achieves these benefits in a cost-effective manner. There is an evidence indicating a break-even point in around 3 to 4 years (EASA, 2018) provides. Pilots who benefited from an evidence-based training are better equipped to manage unforeseen situations, as the methodology focuses on developing competencies that can be applied to different situations rather than mastering how to respond to defined scenarios. It is possible to classify real aviation scenarios in terms of non-technical competencies and filter useful incident reports for design and enrichment of these training scenarios (Dapica and Peinado, 2021).

There are many methods to evaluate training performance. The training and supervision program provide service providers with information guiding its development and implementation (Grunberg et al., 2020). The pragmatic nature of the training and the safe learning environment is considered important to the development of skills and confidence as a supervisor (Harvey et al., 2020). Use of a structured and comprehensive travel training program improves overall travel skills needed for public transportation (Pfeiffer et al., 2020). Evidence for effectiveness of organizational travel plans and other transport interventions which address health and sustainability goals in transport are lacking (Macmillan et al., 2013).

Promotion of eco-driving training in transportation companies could bring a significant reduction in fuel expenses and CO2 emissions (Zavalko, 2018). Coaching is more effective for the improvement of the subjects’ eco-driving level than education alone (Wu et al., 2018). Contemporary education requires well developed basic skills and competencies such as interpersonal communication, teamwork, and leadership (Nutov et al., 2021). Through the extremely randomized tree machine learning technique, degree type and commissioning source are found to be the most important features in determining pilot candidate success (Jenkins et al., 2022).

It may be possible to improve foundational situation awareness skills using deliberate practice, but that further research is required to understand strategies for building higher-level situation awareness skills in evaluation (Mason, 2020). Topic-related knowledge and positive attitudes toward elements of the new law support the course participants’ change-related self-efficacy (Zala-Mezö et al., 2019). Flexible logic model for travel training programs helps scholars and practitioners design and conduct actual evaluation studies. It contributes to theory-driven program evaluation and travel training programs (Park et al., 2016). The high cost of training and the tendency of airlines to shift financial obligations for training to students make the hiring of female pilots difficult in terms of gender inequality (Marintseva et al., 2022).

Evidence-based policy making is an alternative for road safety stakeholders to support science-based policymaking (Papadimitriou and Yannis, 2014). Long-term information on workplace risks has a greater effect and lowers accident rates more than training activities (Mariscal et al., 2019). A gaze-based intervention using video-based, expert commentary, and 3 M (mistake, mitigation, mastery) training to instruct visual flight rule student pilots on an instrument cross-check mitigate the risk of losing aircraft control (Ahmadi et al., 2022).

Primary health care nurses can lead practice change and promote the timely diagnosis and management of dementia in general practice. Training programs of this kind that build knowledge, confidence, awareness and skills should be made available to the primary care nursing workforce (Islam et al., 2020). Evaluators of a newly developed mentor's training program found most of the program's content and resources valuable (Jackson, 2020). There are significant relationships among training effectiveness, work environment, and task performance (Carlisle et al., 2019). Motivation to transfer is often examined in training evaluation studies and reflects the trainee's willingness to apply new knowledge and skills acquired from training to the job (Gegenfurtner and Quesada-Pallarès, 2022).

Practice training reduced participants’ takeover response times compared to paper-based and video-based training (Sahaï et al., 2021). Practical video training program is useful to improve teachers’ type 1 diabetes (T1D)-related self-confidence in their skills and knowledge for their children (Tournilhac et al., 2020). Conscientiousness and self-confidence increase pilot training success but neuroticism decreases (Breuer et al., 2023). After the coaching for life skills online training program, both experimental and waitlist group participants show the increases in mean scores observed (Camiré et al., 2020).

Continuous techno-training can help in developing techno-efficacy directly as well as indirectly through fostering techno-expectancy and suppressing techno-stress (Rayburn et al., 2021). Practice-based learning environments can foster teachers’ professional vision of critical incidents in classroom through case-based learning from video as well as problem-based learning (LuiseBarth et al., 2019). Biofeedback training can be an effective intervention for aviation authorities and airline operators to develop peer support programs for pilots to increase psychological resilience and well-being (Zhang et al., 2022). The remote pilot competences (RPCs) for initial training were related to cognitive abilities and theory knowledge, but flight skills, personality aspects, and interpersonal skills were relevant for recurrent training (Schmidt et al., 2022).

Although the excellence of the new training system, EBT, is recognized by the global aviation industry, the reality is that airlines that should adopt and operate it for financial reasons, or because of pilots’ resistance, are hesitant to invest in the new training system. This is a general challenge faced by every introduction of a new system, as can be seen in the history of training systems development. This paper identifies the effects of EBT for airlines operating for companies considering the introduction of EBT.

Theoretical background

There are many studies on airline safety. It is known that organization is the most important dimension in safety management system (SMS), which begins with policies that convey to all staff the top managers’ vision on safety (Hsu et al., 2010). Safety investment as a mediating variable between the financial health of an airline and its accident propensity and simultaneously account for the reverse effects of accident propensity on safety investment (Wang et al., 2013). Long-haul transport companies may use individual truck driver characteristics in their training and selection procedures to meet operational safety and productivity objectives (Vries et al., 2017).

Despite differences in different areas, studies continue to verify or expand in various industries based on the IS success model of DeLone and McLean (1992). In this study, we review the preceding studies that were conducted based on the IS success model and present a new model based on the IS success model to empirically verify the effectiveness of the EBT program.

Based on existing studies related to the introduction and utilization of information systems, DeLone and McLean (1992) approached the topic in a more specific and empirical way and presented a multidimensional model that could be applied universally and to many environments. In the IS success model, six variables were derived that could measure the success of the information system: system quality, information quality, use, user event, individual effect, and organization effect. The IS success model said that each of these six variables acts more interrelation ally than independently. The quality of the system and the quality of the information affect usage and user satisfaction in isolated or combined ways, and usage and user satisfaction have positive and negative effects on each other (Ojo, 2017; Petter et al., 2008). In addition, the model was constructed in which usage precedes user satisfaction and ultimately results in organizational impact (Seo et al., 2003).

This model has been expanded or detailed by many researchers, and based on these studies, DeLone and McLean proposed an improved information system success model 10 years later as Figure 1 shows (DeLone and McLean, 2003). Under the newly proposed success model, the quality of service in Pitt et al. (1995) was included at the level of the change in the user's computing environment to accommodate the change in the situation in which the organization that operates the information system concurrently controls information services (Kim, 2016). The following are the models of the newly proposed model.

The IS Success Model of DeLone and McLean has been adopted not only in the field of information systems but also in various fields such as e-Commerce, repurchase intention, e-Government and e-Learning. Based on the existing DeLone and McLean IS Success Model, suggesting customer e-commerce satisfaction (CES) as a dependent variable to assess the success of e-commerce is suggested (Molla and Licker, 2001). the The application of their existing IS success model to assess the success of e-commerce systems is studied (DeLone and McLean, 2004). The new dimensions related to system quality, service quality, perceived benefits and perceived costs in international e-commerce are designed, while studying the factors related to the success and credit and usage of international e-commerce (Cui et al., 2018).

Based on the IS success model of DeLone and McLean, models were also studied to assess customers repurchase intentions. The relationship between the repurchase intention by introducing justice and trust elements into existing models is revealed (Fang et al., 2011; Hsu et al., 2014).

The performance of e-Government was also evaluated based on the IS success model of DeLone and McLean from the view of citizens (Rana et al., 2015; Veeramootoo et al., 2018; Wang and Liao, 2008). The IS success model of DeLone and McLean was used to demonstrate the causal relationship between quality attributes and usage intention, usage, and user satisfaction in the use of a cloud office (Yoon and Kim, 2015).

Research model

IS success model consists of system quality and information quality. As a training method, EBT has the similar aspects. This is, the EBT performance could be affected by the training system and the information system. That is the reason why this study adopts IS success model to explain the success factors of EBT.

In this study, a research model based on the IS success model is proposed for the new pilot training program, EBT, which is operated by airlines, to find out the impact of the quality of training systems and training information on pilot satisfaction as a training program user and personal capacity improvement. Figure 2 shows a research model on the success factors of evidence-based training for pilots.

Training system quality and trainee satisfaction

DeLone and McLean (2003) were able to measure system quality by the units of ease-of-use, functionality, reliability, flexibility, data quality, portability, integration, and importance, as a measure of the quality and performance of the information system itself. DeLone and McLean (1992) also said that satisfaction is the most preferred indicator for measuring the effectiveness of the information system in reporting responses to the information system. This study investigated the satisfaction level of trainees (pilots).

In this study, the training system quality was measured by the appropriateness of training topics, the ease of learning, and the reliability of the instructor, who directs the training. In EBT, the instructor considers the quality of his or her material as a facilitator, not as a past instructor or examiner, and is required to receive regular Interrater Reliability (IRR) training to ensure reliability among the instructors. The following assumptions were established because the characteristics of these EBT programs determined that the user (pilot) would be highly satisfied.

Hypothesis 1: Training system quality will have a positive effect on trainee satisfaction.

Training information quality and trainee satisfaction

DeLone and McLean (1992) referred to information quality as a measure of the results of an information system and said it was judged based on the characteristics of the information system results, such as accuracy, output timeliness, reliability, completeness, relevance, precision, and currency.

The EBT program collected and analyzed years of aviation-related data to select appropriate training topics and published a book titled “Data Report for Evidence-Based Training (2013)” (IATA, 2013a) and established a duration of training topics on this basis. Non-competency-related topics were also included in the training of human factors that contribute to flight accidents, which can be important in securing pilot training usability. Since the quality of this information is also believed to help the satisfaction of the trainees (pilot), the hypothesis was established as follows.

Hypothesis 2: Training information quality will have a positive effect on trainee satisfaction.

Training system quality and perceived competency improvement

DeLone and McLean (2003) demonstrated a statistically direct association between system quality and individual impact. On the other hand, the index impact is measured by the quality of the work environment and job performance. In this study, system quality was considered training system quality, and individual impact was perceived competency implementation by individual trainees.

In addition, because it is difficult to objectively assess the pilots’ improvement in their ability to do so, they have been assessed to the extent that they are evaluated. Since the quality of the training system is also determined to correlate positively with perceived capacity improvement, the following hypothesis was established.

Hypothesis 3: Training system quality will have a positive effect on the perceived competency implementation.

Training information quality and perceived competency improvement

DeLone and McLean (2003) found that many studies of the relationship between information quality and individual impact showed a large association between the two variables. The quality of the information could also be measured in terms of decision-making performance, job effectiveness, and quality of work.

The EBT program aims to improve pilots’ resiliency through training to make the right decisions in the event of unexpected abnormalities during flights, thereby increasing the safety of flight operations. For this reason, the hypothesis was set as follows, as it was determined that there would be a positive relationship between the quality of training information and the enhancement of the ability of the pilot.

Hypothesis 4: Training information quality will have a positive effect on perceived competency implementation.

Research methodology

The purpose of this study is to verify whether evidence-based training is effective for airline pilot. The effectiveness of evidence-based training was investigated for pilot. The instrument is adapted from DeLone and McLean (1992) which measures the information system success. The instrument is at the Appendix. Each item of the survey was measured using the Likert's 5-point scale. 1 was strongly disagree, and 5 was strongly agree.

In this study, five months data of EBT Training Satisfaction Survey were used by an airline currently operating the EBT program, which conducted a survey. The survey was conducted directly on pilots who had completed regular training and collected data monthly. A total of 1003 people were subjected to the study, including 557 captains and 446 assistant captains of each model who responded to the survey conducted after regular training. Demographics of survey respondents as followings as Table 2.

Table 2.

Demographics of survey respondents.

Classification	Classification	Number of respondents	Ratio
Rank	Captain	557	55.53%
Rank	Co-pilot	446	44.47%
Flight model	A320	240	23.93%
	A330	148	14.76%
	A350	131	13.06%
	A380	107	10.67%
	B747	169	16.85%
	B767	65	6.48%
	B777	143	14.26%

The proportion of captain and co-pilot was similar to 55.53% and 44.47%. The proportion of the flight model was in the order of B747 (16.85%), A330 (14.76%), and B777 (14.26%).

The survey questionnaire was delivered to the pilots who participated in the training online, and the pilots also responded to the questionnaire online. The survey questionnaire consisted of items that investigated the response of the airline pilot to the evidence-based training based on the question items presented in the IS success model. The system quality and information quality of training were investigated by consisting of independent variables, and the satisfaction of trainees and perceived competence improvement were measured as dependent variables. Independent and dependent variables were measured at Likert's 5-point-scale, respectively.

In order to measure the net effect of each variable, variables such as position, flight model, and training period were considered in the model as control variables. Control variables, such as position, flight model, and training period are coded as dummy variables. The effects of control variables are excluded in the interpretation of research results.

The total number of pilots working for the airline was 1003 including 557 captains and 446 co-pilots. The survey method was conducted through satisfaction surveys conducted on pilots after regular training. Survey participants are asked the questions of the training system quality and the training information quality after the training.

This research does not apply the before-and-after study, but a cross-sectional study. This study measures the relationships among variables after the training. When the purpose of the study is to measure only the effectiveness of evaluation-based training, before-and-after study may be a more suitable methodology. In this study, a cross-sectional study would be more appropriate to discover the effectiveness of evaluation-based training and the influencing factors that affect the effectiveness of evaluation-based training.

The results of the survey were analyzed using the SPSS (V25) statistical package. The research model in this study has two by two relationships. The research model has single relationship, not a path. That is why regression analysis is selected as a statistical method, rather than other alternative methods. Two regression analysis was performed to measure the causal relationship for two independent variables and two dependent variables.

Results

Regression results on trainee satisfaction

To verify the effect of Training system quality and Training information quality on Trainee satisfaction, a multiple regression analysis was performed. Table 3 shows that the regression model was statistically significant (F = 935.756, p < .001), and the explanation power of the regression model was about 65.2% (adjusted R² is 65.1%).

Table 3.

Results of regression: H1 and H2.

Dependent variables	Independent variables	B	S.E.	β	t	P
Trainee satisfaction	Constant	0.206	0.089		2.314	0.021
	Training system quality	0.521	0.031	0.493	16.695	0.000
	Training information quality	0.405	0.033	0.362	12.249	0.000
F = 935.756(p < .001), R² = .652, adjR² = .651

The results of the validation of significance of the regression coefficients showed that both Training system quality and Training information quality had a significantly positive effect on the Trainee satisfaction. In other words, it was assessed that the higher the Training system quality and Training information quality, the higher the training system perceived by the pilot. When comparing the size of the standardized coefficients, it is found that the Training system quality (β = .493) has a greater influence on the Trainee satisfaction than the Training information quality (β = .362) does.

Therefore, it was found that hypothesis 1 that Training system quality affects Trainee satisfaction and hypothesis 2 that Training information quality affects Trainee satisfaction were accepted. It can be concluded that increasing the training system quality and training information quality can increase the trainee satisfaction, which can enhance the educational effect.

Regression results on perceived competency improvement

To verify the effect of Training system quality and Training information quality on Perceived competency improvement, a multiple regression analysis was performed. Table 4 show that the regression model was statistically significant (F = 956.489, p < .001), and the explanation power of the regression model was about 65.7% (adjusted R² is 65.6%).

Table 4.

Results of regression: H3 and H4.

Dependent variables	Independent variables	B	S.E.	β	t	P
Perceived competency improvement	Constant	0.038	0.092		0.420	0.674
	Training system quality	0.534	0.032	0.488	16.623	0.000
	Training information quality	0.431	0.034	0.371	12.650	0.000
F = 956.489(p < .001), R² = .657, adjR² = .656

The results of the validation of significance of the regression coefficients showed that both Training system quality and Training information quality had a significantly positive effect on the Perceived competency improvement. In other words, it was assessed that the higher the Training system quality and Training information quality, the higher the competency improvement perceived by the pilot. When comparing the size of the standardized coefficients, it is found that the Training system quality (β = .488) has a greater influence on the Perceived competency improvement than Training information quality (β = .372) does. Empirical results show that the explanation powers of independent variables are not decreased even though there is a concern on the slightly strong positive relationship between independent variables.

Therefore, it was found that hypothesis 3 that Training system quality affects Perceived competence improvement and hypothesis 4 that Training information quality affects Perceived competence improvement were accepted. It can be concluded that increasing the training system quality and training information quality can increase the perceived competence improvement, which can be interpreted to be a training objective. Based on the above findings, the final analysis of the study model is shown in Figure 3.

Conclusion and discussion

The purpose of this paper is to identify the need to introduce a new training system, the EBT model, and to assess and verify its performance by operating airlines to assist in decision making by airlines considering the introduction of EBT. This study first explains the development process of pilot training system and explores the need to introduce a new system. Based on the IS success model, a new analysis model is proposed, and the results of the satisfaction survey conducted on pilots are analyzed through multiple regression analyses. The main results of this study are as follows.

First, from the beginning of the jet era, the history of the development of pilot training system in commercial aviation industry and its effects have been examined to identify the need for continued development of training program. To ensure flight safety for fast-developing aviation technology and rapidly changing flight environments, pilots are required to develop their capabilities. In particular, the importance of training methods to enhance non-technical performance related to pilots’ human factors, as well as technical performance, which was valued in existing training, has emerged. As part of efforts to develop pilot non-technical performance, pilot training systems have developed into CRM, LOFT, AQP, ATQP, etc., and a new training system called EBT has been created in the ongoing line of development of such training systems.

Second, based on the IS success model of DeLone and McLean, a new model was presented to evaluate the success of the new training program, EBT. Based on this analysis model, the results of the survey conducted were analyzed through multiple regression analysis to confirm the validity of this model, and the pilots who were trained were also satisfied with EBT.

The results of this study demonstrated that evidence-based training is effective in terms of trainee satisfaction and perceived competency improvement. It was demonstrated that such trainee satisfaction and perceived competency improvement are affected by the information quality and service quality of education. These training events can be seen to improve as the training system quality and training information quality increase. In addition, the higher the training system quality and the training information quality, the better the pilots’ abilities are. This indicates that the quality control efforts of the company performing the EBT are also important for its success.

These results confirmed the effectiveness of evidence-based training in the education of aircraft pilots, and showed the same results as existing academic evidence, which was proved through empirical analysis. These results can be expected to enable customized education and education to adapt to changes in pilot education in the future.

Moreover, by measuring the effectiveness of evidence-based training by trainee satisfaction and perceived competency improvement, it is possible to confirm the effectiveness of trainee's psychological satisfaction and performance. In addition, by revealing that information quality and service quality are the factors that affect these performances, it is expected that it can be used as an important reference for the content and method of education delivered to trainee in terms of education content and method.

The results of this study found that evidence-based training helps airline pilots training. These results are estimated to be applicable not only to airline pilot training but also to education and training in various fields. In addition, for evidence-based training to be effective, it is necessary to design a training program considering that the system quality and information quality of training affect trainees’ satisfaction and perceived competence improvement.

From the view of academics, this study identifies the IS success model by DeLone and McLean could be applied not only to the information system field, but also to other industries, and confirms the potential for its use in evaluating the success of pilot training systems.

From the view of practice, introducing a new system requires a big decision for management. Moreover, when introducing a new system in the presence of an existing system, the need and the logical basis for predicting performance are essential. This paper contributes a positive basis for EBT success.

Limitations & future studies

This study was based on secondary data, i.e., the results of a training satisfaction survey conducted by an airline company, which limited the question items. Measurements from the limited question items provided in the secondary data have not been made to look at various aspects that may affect the satisfaction of the training. If future measurement variables are used, for example, the differences in satisfaction between the captain and the co-pilot, or the type of aircraft, a further study would provide a closer look at the factors necessary for the EBT program to be successfully established.

This study tests hypotheses of the impact of system and information qualities on training performance. It can measure more objectively when the before-and-after study is applied. However, this research uses the cross-sectional study instead, due to the practical issue that it is hard to measure respondents before the training. The future study would have more objective results in case of applying the before-and-after study.

In terms of measurement of variables, this study measures single indicator for each variable due that this study uses the secondary data. It could be a problem for objective measurement. In the future studies, it would be objective to measure constructs with multiple indicators.

The role of the legal/regulatory system in the attribution of blame could be developed a bit further in the context of pilot training and accident investigations. In this study, only the training method is examined even though the legal or regulatory issue is also important.

Footnotes

ORCID iD

Jin Ki Kim

About the authors

Eui Song Hwang is a captain of the A321 fleet at Asiana Airlines. He received his MS in Aviation Business from Korea Aerospace University. He has a BS from Hankuk University of Foreign Studies in Seoul, Korea. He had been working in a Flight Crew Training Team as a training program developer and a fleet training manager for many years, and he is now an instructor captain of the A321. His interests are pilot training program development, and data analysis in the aviation industry.

Jin Ki Kim is a professor at the school of business in the Korea Aerospace University. He received his PhD in Management Information Systems from the State University of New York at Buffalo. He has an MS and a BS from Hanyang University at Seoul in Korea. Prior to joining doctoral program, he worked at the Korea Information Society Development Institute (KISDI) as a research fellow. His interests are ICT business strategy, telecommunications policy and management, digital convergence and eco-system, media management and strategy, ICT payoff and emergency management systems. His research has appeared in International Journal of Mobile Communications, Journal of Air Transport Management, Research in Transportation Business & Management, Information Development, Technology in Society, Computers in Human Behavior, International Journal of Tourism Research, Electronic Commerce Research and Applications, Information Systems Frontier, Communications of the Association for Information Systems (CAIS), and Decision Support Systems, and the following conferences: HICSS, AMCIS, TPRC and ITS.

Appendix: Questionnaire

Q1: Training System Quality: Has the instructor conducted training and briefed/debriefed on pilot competency in line with the EBT concept?

Q2: Training Information Quality: Has your company sufficiently communicated and understood about your EBT-related concepts?

Q3: Trainee Satisfaction: Do you think the EBT concept of training is an improvement over the previous system?

Q4: Perceived Competency Implementation: Do you believe that EBT concept training is structured to assist individuals in developing their capabilities in keeping with changing flight environments?

References

Ahmadi

Romoser

Salmon

(2022) Improving the tactical scanning of student pilots: a gaze-based training intervention for transition from visual flight into instrument meteorological conditions. Applied Ergonomics 100: 103642.

Allianz_Global_Corporate_&_Specialty (2014) Global Aviation Safety Study: Allianz Global Corporate & Specialty.

Breuer

Ortner

Gruber

, et al. (2023) Aviation and personality: do measures of personality predict pilot training success? Updated meta-analyses. Personality and Individual Differences 202: 111918.

CAA (2016) Standards Document 80_Version 1. Sejong, South Korea: Alternative Training and Qualification Programme (ATQP) Industry Guidance, Civil Aviation Authority (CAA).

Camiré

Kendellen

Rathwell

, et al. (2020) Evaluating the coaching for life skills online training program: a randomised controlled trial. Psychology of Sport & Exercise 48: 101649.

Carlisle

Bhanugopan

D’Netto

(2019) Enhancing task performance through effective training: the mediating role of work environment and moderating effect of non-mandatory training. Journal of Business Research 104: 340–349.

Cui

Mou

Cohen

, et al. (2018) Understanding IS success model and valence framework in sellers’ acceptance of cross-border e-commerce. WHICEB Proceedings, 39.

Dapica

Peinado

(2021) Towards a semantic knowledge base for competency-based training of airline pilots. Procedia Computer Science 192: 1208–1217.

DeLone

McLean

(1992) Information systems success: the quest for the dependent variable. Information Systems Research 3(1): 60–95.

10.

DeLone

McLean

(2003) The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems 19(4): 9–30.

11.

DeLone

McLean

(2004) Measuring e-commerce success: applying the DeLone & McLean information systems success model. International Journal of Electronic Commerce 9(1): 31–47.

12.

EASA (2018) Update of ORO.FC – Evidence-based Training Subtask. Köln, Germany: European Aviation Safety Agency (EASA).

13.

ECA (2019) Implementation of Evidence Based Training (EBT) – Position & Detailed Comments in Response to NPA 2018-07. Brussels, Belgium: European Cockpit Association.

14.

Fang

Chiu

Wang

(2011) Understanding customers’ satisfaction and repurchase intentions: an integration of IS success model, trust, and justice. Internet Research 21(4): 479–503.

15.

Gegenfurtner

Quesada-Pallarès

(2022) Toward a multidimensional conceptualization of motivation to transfer training: validation of the transfer motivation questionnaire from a self-determination theory perspective using bifactor-ESEM. Studies in Educational Evaluation 73: 101116.

16.

Grunberg

Dennis

C-L

Costa

, et al. (2020) Development and evaluation of an online infertility peer supporter training program. Patient Education and Counseling 103(5): 1005–1012.

17.

Harvey

Spurr

Sidebotham

, et al. (2020) Describing and evaluating a foundational education/training program preparing nurses, midwives and other helping professionals as supervisors of clinical supervision using the role development model. Nurse Education in Practice 42: 102671.

18.

Helmreich

Merritt

Wilhelm

(1999) The evolution of Crew Resource Management training in commercial aviation. International Journal of Aviation Psychology, 9(1): 19–32.

19.

Hembroff

Arbuthnott

Krätzig

(2018) Emergency response driver training: dual-task decrements of dispatch communication. Transportation Research Part F: Traffic Psychology and Behaviour 59(A): 222–235.

20.

Hsu

Y-L

W-C

Chen

K-W

(2010) Structuring critical success factors of airline safety management system using a hybrid model. Transportation Research Part E: Logistics and Transportation Review 46: 225–235.

21.

Hsu

Chang

Chu

, et al. (2014) Determinants of repurchase intention in online group-buying: the perspectives of DeLone & McLean IS success model and trust. Computers in Human Behavior 36: 234–245.

22.

IATA (2013a) Data Report for Evidence-Based Training. International Air Transport Association.

23.

IATA (2013b) Evidence-Based Training Implementation Guide. Montreal, Canada: International Air Transport Association.

24.

IATA (2018) IATA Annual Review 2018. Montreal, Canada: International Air Transport Association (IATA).

25.

ICAO (2013) ICAO Doc 9995, Manual of Evidence-Based Training. Montreal, Canada: International Civil Aviation Organization.

26.

ICAO (2015) ICAO Doc 9868, PANS-TRG. Montreal, Canada: International Civil Aviation Organization (ICAO).

27.

ICAO (2018) ICAO Safety Report. Montreal, Canada: International Civil Aviation Organization (ICAO).

28.

Islam

Parkinson

Burns

, et al. (2020) A training program for primary health care nurses on timely diagnosis and management of dementia in general practice: an evaluation study. International Journal of Nursing Studies 105: 103550.

29.

Jackson

(2020) Evaluation of a mentor’s training program for nurse practitioners. The Journal for Nurse Practitioners 16(4): 286–289.

30.

Jenkins

Caballero

Hill

(2022) Predicting success in United States air force pilot training using machine learning techniques. Socio-Economic Planning Sciences 79: 101121.

31.

Kelly

Efthymiou

(2019) An analysis of human factors in fifty controlled flight into terrain aviation accidents from 2007 to 2017. Journal of Safety Research 69: 155–165.

32.

Kim

(2016) A Study on the Factors Influencing the Perceived IT Investment Intention : An Analysis of Differences in Service Types and Job Ranks . PhD Dissertation, Dankook University, Yongin City. http://www.riss.kr/link?id=T14198520.

33.

Lauber

Foushee

(1981) Guidelines for Line-Oriented Flight Training. Volume 1. Moffett Field, CA, United States: the NASA/Industry Workshop.

34.

LuiseBarth

Piwowar

Kumschick

, et al. (2019) The impact of direct instruction in a problem-based learning setting. Effects of a video-based training program to foster preservice teachers’ professional vision of critical incidents in the classroom. International Journal of Educational Research 95: 1–12.

35.

Macmillan

Hosking

Connor

, et al. (2013) A Cochrane systematic review of the effectiveness of organisational travel plans: Improving the evidence base for transport decisions. Transport Policy 29: 249–256.

36.

Marintseva

Mahanecs

Pandey

, et al. (2022) Factors influencing low female representation in pilot training recruitment. Transport Policy 115: 141–151.

37.

Mariscal

López-Perea

López-García

, et al. (2019) The influence of employee training and information on the probability of accident rates. International Journal of Industrial Ergonomics 72: 311–319.

38.

Martins

(2021) Employee training and firm performance: evidence from ESF grant applications. Labour Economics 72: 102056.

39.

Mason

(2020) Practice makes better? Testing a model for training program evaluators in situation awareness. Evaluation and Program Planning 79: 101788.

40.

Molla

Licker

(2001) E-commerce systems success: an attempt to extend and respecify the Delone and MacLean model of IS success. Journal of Electronic Commerce Research 2(4): 131–141.

41.

Moro

(2019) Evidence Based Training: time to change traditional pilot training. Data Science in Aviation.

42.

Morshed

Ballew

Elliott

, et al. (2017) Evaluation of an online training for improving self-reported evidence-based decision-making skills in cancer control among public health professionals. Public Health 152: 28–35.

43.

Morton

Colenso-Semple

Phillips

(2019) Training for strength and hypertrophy: an evidence-based approach. Current Opinion in Physiology 10: 90–95.

44.

Nutov

Gilad-Hai

Maskit

(2021) Complementary assessment in teacher and school leadership training: necessity, conceptualization and validation. Studies in Educational Evaluation 71: 101070.

45.

Ojo

(2017) Validation of the DeLone and McLean information systems success model. Healthcare Informatics Research 23(1): 60–66.

46.

Papadimitriou

Yannis

(2014) Needs and priorities of road safety stakeholders for evidence-based policy making. Transport Policy 35: 286–294.

47.

Park

Welch

Sriraj

(2016) An integrative theory-driven framework for evaluating travel training programs. Evaluation and Program Planning 59: 7–20.

48.

Petter

DeLone

McLean

(2008) Measuring information systems success: models, dimensions, measures, and interrelationships. European Journal of Information Systems 17: 236–263.

49.

Pfeiffer

Sell

Bevans

(2020) Initial evaluation of a public transportation training program for individuals with intellectual and developmental disabilities: short report. Journal of Transport & Health 16: 100813.

50.

Pitt

Watson

Kavan

(1995) Service quality: a measure of information systems effectiveness. MIS Quarterly 19(2): 173–187.

51.

Rana

Dwivedi

Williams

, et al. (2015) Investigating success of an e-government initiative: validation of an integrated IS success model. Information Systems Frontiers 17(1): 127–142.

52.

Rayburn

Badrinarayanan

Anderson

, et al. (2021) Continuous techno-training and business-to-business salesperson success: how boosting techno-efficacy enhances sales effort and performance. Journal of Business Research 133: 66–78.

53.

Rosen

Ruzek

Karlin

(2017) Evidence-based training in the era of evidence-based practice: challenges and opportunities for training of PTSD providers. Behaviour Research and Therapy 88: 37–48.

54.

Sahaï

Barré

Bueno

(2021) Urgent and non-urgent takeovers during conditional automated driving on public roads: the impact of different training programmes. Transportation Research Part F: Traffic Psychology and Behaviour 81: 130–143.

55.

Schmidt

Schadow

Eißfeldt

, et al. (2022) Insights on remote pilot competences and training needs of civil drone pilots. Transportation Research Procedia 66: 1–7.

56.

Seo

Yoon

Kim

(2003) Correlation analysis of impacting factors on the IT investment and performance: extended IS success model. Entrue Journal of Information Technology 2(2): 101–114.

57.

Tomczak

Lanzo

Aguinis

(2018) Evidence-based recommendations for employee performance monitoring. Business Horizons 61(2): 251–259.

58.

Tournilhac

Dolladille

Armouche

, et al. (2020) Evaluation of a new training program to reassure primary school teachers about glucagon injection in children with type 1 diabetes during the 2017–2018 school year. Archives de Pédiatrie 27(4): 212–218.

59.

Veeramootoo

Nunkoo

Dwivedi

(2018) What determines success of an e-government service? Validation of an integrative model of e-filing continuance usage. Government Information Quarterly 35(2): 161–174.

60.

Vries

Koster

Rijsdijk

, et al. (2017) Determinants of safe and productive truck driving: empirical evidence from long-haul cargo transport. Transportation Research Part E: Logistics and Transportation Review 97: 113–131.

61.

Wang

Y-S

Liao

Y-W

(2008) Assessing eGovernment systems success: a validation of the DeLone and McLean model of information systems success. Government Information Quarterly 25(4): 717–733.

62.

Wang

Hofer

Dresner

(2013) Financial condition, safety investment and accident propensity in the US airline industry: a structural analysis. Transportation Research Part E: Logistics and Transportation Review 49: 24–32.

63.

Weitzel

(1992) A turning point in aviation training: The AQP mandates crew resource management and line operational simulations. Journal of Aviation/Aerospace Education & Research 3(1): 2.

64.

Wilfley

Fitzsimmons-Craft

Eichen

, et al. (2018) Training models for implementing evidence-based psychological treatment for college mental health: a cluster randomized trial study protocol. Contemporary Clinical Trials 72: 117–125.

65.

Zhao

Rong

, et al. (2018) The effectiveness of eco-driving training for male professional and non-professional drivers. Transportation Research Part D: Transport and Environment 59: 121–133.

66.

Yoon

Kim

(2015) A study on the use activation of the cloud office with focus on DeLone and McLean IS success model. Journal of Information Technology Services 14(2): 289–314.

67.

Zala-Mezö

Raeder

Strauss

N-C

(2019) More knowledge helps implement change: evaluation of a training program during educational reform. Studies in Educational Evaluation 61: 105–111.

68.

Zavalko

(2018) Applying energy approach in the evaluation of eco-driving skill and eco-driving training of truck drivers. Transportation Research Part D: Transport and Environment 62: 672–684.

69.

Zhang

W-C

Andrews

(2022) Applying psychophysiological coherence training based on HRV-biofeedback to enhance pilots’ resilience and wellbeing. Transportation Research Procedia 66: 49–56.