Abstract
Complex domains require cognitive work for which current approaches to training may be ill-suited. To improve training for cognitive work, Klein and Baxter have proposed Cognitive Transformation Theory (CTT), a learning theory that characterizes sensemaking processes as essential to the development of expertise. The objectives of this research were to compare CTT with the instructional strategies of two expert air traffic control instructors to evaluate the relevance of CTT’s four teaching practices, propose refinements to CTT, and identify potential instructional strategies to serve as guidance for the application of CTT. Data were collected using cognitive task analysis methods, including course observation, artifact examination, and knowledge elicitation with two instructors and seven of their students. Data were coded using categories derived from theory and patterns emergent within the data. Results suggest that many of the instructional strategies used were consistent with the teaching practices of CTT and that learning was aligned with the active sensemaking claims of CTT. An integrated set of instructional strategies and a few refinements to CTT are advanced to further its application to training in complex domains. Although this set of strategies may benefit current training practices, further research is needed to evaluate their effectiveness.
Keywords
In this age of information and networks, many work domains require humans to process and make sense of large quantities of dynamic information, navigate complex system infrastructures, coordinate with multiple stakeholders, and in general consult and consider a potentially vast array of variables when solving complex problems and making decisions. These demands translate into significant cognitive work and call for a capacity to flexibly adapt to the dynamics and complexities of that work. Hoffman, Feltovich, Fiore, Klein, and Zeibell (2009) suggest that current approaches to training may be inadequate for the cognitively demanding work many people currently are required to perform, a notion echoed in the writings of Klein and Baxter (2006, 2009). To improve training so that it aligns better with the cognitive demands of most modern work, Klein and Baxter proposed Cognitive Transformation Theory (CTT), a learning theory that claims sensemaking activities are essential for developing expertise that is adaptive and well suited for complex cognitive work. More specifically, CTT asserts that there are a number of tacit forms of knowledge required for developing adaptive expertise for complex cognitive work and these forms of knowledge are likely not being facilitated by current approaches to education and training.
This article attempts to bridge the gap between theory and practice by empirically investigating the real-world instruction of novices and the strategies used by expert instructors in the complex cognitive work domain of air traffic control (ATC). Broadly, the aim is to evaluate the relevance of CTT’s four core teaching practices and to identify specific instructional strategies that can serve as examples of and guidance for further empirical evaluation and application of CTT to training and educational systems. We first review the nature of adaptive expertise, given that the study of individuals with such expertise, to some degree, motivated and inspired the development of CTT. Then, we provide an overview of CTT and its core teaching practices. Lastly, before detailing our research approach and objectives, we review the Data/Frame Theory of Sensemaking (e.g., Klein, Moon, & Hoffman, 2006b; Sieck, Klein, Peluso, Smith, & Harris-Thompson, 2007) and similar theories of learning and skill acquisition, given that the engagement of sensemaking processes in students and trainees is an essential component of CTT.
Adaptive Expertise
Decades of research support the notion that as people develop expertise, they tend to acquire a deep conceptual understanding and a contextualized knowledge base that helps them focus their attention on meaningful cues, detect patterns and pattern shifts with an understanding of their implications, anticipate and take proactive action, evaluate potential consequences of choices, and more (see Bransford, Brown, & Cocking, 2000; Ericsson, 2009; Ericsson, Charness, Feltovich, & Hoffman, 2006; Ericsson & Ward, 2007; Hoffman et al., 2014, for reviews). Although these generalizations often hold true, researchers have observed that an individual’s expertise can be characterized along a continuum of routine to adaptive expertise (e.g., Bransford et al., 2000; Chi, 2006; Hatano & Inagaki, 1986; Hoffman et al., 2009; Hoffman et al., 2014; Schwartz, Bransford, & Sears, 2005). Routine expertise, on the one hand, refers to the expert, fluent execution and knowledge of a domain’s rules and procedures; though, these are often applied in a rigid and inflexible manner with lack of recognition for exceptions to the rules. By contrast, adaptive expertise refers to a type of expertise that supports resourcefulness, problem solving, an ability to draw on one’s knowledge base to respond to novel and ambiguous situations, and further, the ability to understand when procedures need to deviate from the normal or when traditional boundaries can be crossed.
Given the ill-structured and increasingly dynamic nature of complex cognitive work domains, a need exists, now more than ever, for experts who are able to apply their knowledge flexibly and adaptively (e.g., Feltovich, Spiro, & Coulson, 1993; Hoffman et al., 2009; Hoffman et al., 2014; Van Merriënboer, Jelsma, & Paas, 1992). CTT, a theory inspired by decades of work in real-world complex domains with a focus on expertise, has been proposed as one solution to address this need (Hoffman et al., 2009; Hoffman et al., 2014); though, specific applications of this theory are still quite scarce (e.g., Sottilare & Goldberg, 2012).
CTT
Klein and Baxter (2006, 2009) describe a tendency for people to think of learning as the passive accumulation of both factual knowledge and knowledge of routines and procedures, the use of which becomes increasingly fluid with practice. They imply that teaching and training practices that rely on this form of learning as their objective are inadequate because they do not focus on or adequately facilitate expertise acquisition. Rather, to the extent these teaching practices contribute to expertise, it is the routine form of expertise. With CTT, Klein and Baxter promote a different path to expertise and propose teaching practices that support it. According to CTT, the path to adaptive expertise is one of learning not only factual and procedural knowledge but also the ability to recognize familiar patterns, the skills to make perceptual discriminations, and—critically—the interconnections among these forms of explicit and tacit knowledge. In turn, this path to adaptive expertise is further characterized by the ability to modify and tune these forms of knowledge to the specifics of the domain as experience is gained.
Drawing from the theoretical traditions of Kolb (1984) and Dewey (1938), what differentiates CTT from many other cognitive theories of learning is its rejection of traditional information processing approaches to learning and their use of a storehouse metaphor (e.g., learning as accumulation of knowledge; see, e.g., Mayer, 2005). CTT instead emphasizes perpetual mental model development (i.e., learning as a continual integration and refinement of various forms of knowledge) and periodic mental model rejection and even unlearning. Mental models as a construct have an extensive history in cognitive psychology (e.g., Craik, 1943; Gentner & Stevens, 1983; Johnson-Laird, 1983, 2004). In Klein and Baxter’s case, the term mental model is used to refer to the interconnected body of knowledge that forms a representation of the core causal relationships in a domain and allows an individual to both explain and predict how events unfold. This use of the term mental model is similar to many other definitions and constructs (e.g., schemas, templates, scripts, and situation models) with a key difference in CTT being the focus on mental model formation and use as an active, intentional process relying on sensemaking activities (Klein, Moon, & Hoffman, 2006a), rather than passive and automatic outcomes of cumulative experience and information processing mechanisms.
Central to CTT is the notion that to understand mental model development, it is important to acknowledge that when novices develop rudimentary mental models they are regularly subject to simplifications and inaccuracies (e.g., Feltovich et al., 1993; O’Brien & Albrecht, 1992). Therefore, in CTT, Klein and Baxter (2006, 2009) argue that learning and expertise acquisition depend on a process of mental model development, assessment, modification, and attunement in which active sensemaking plays a pervasive role. The main goal of CTT is to help learners continuously remold their mental models. As Bransford, Franks, Vye, and Sherwood (1989) wrote in their article New Approaches to Instruction: Because Wisdom Can’t Be Told, “perhaps wisdom arises from the opportunity to experience changes in our own beliefs and assumptions—changes that help us realize that the ideas and priorities that seem so clear today will probably be modified as a function of new experiences” (p. 494). In support of these probable and valuable modifications, CTT proposes four core teaching practices:
diagnostic assessments that identify flaws in students’ mental models,
learning objectives that emphasize sensemaking and encourage reflection of new learning so that deeper and richer mental models are formed and revised,
practice that incorporates sensemaking to give students experience figuring out what data matter and in what contexts, and
immediate nondisruptive feedback that prompts sensemaking, and support for learning to seek and interpret feedback on one’s own.
These four teaching practices have been subject to extensive research in the learning, training, and education literature. Indeed, a robust body of research literature addresses the timing of feedback (e.g., Marshall, 1995; Mathan & Koedinger, 2005; Shute, 2008), the nature of feedback content (e.g., McKendree, 1990; Shute, 2008; Wulf, McConnel, Gartner, & Schwarz, 2002), strategies for encouraging learners to seek their own feedback (Goodman, Wood, & Hendrickx, 2004; Mathan & Koedinger, 2005), and practice (e.g., Bransford, Franks, Morris, & Stein, 1979; Donovan & Radosevich, 1999; Ericsson, 2006). Nor is the importance of learning objectives a new idea—many researchers have argued the importance of aligning training materials and strategy with prespecified learning objectives (e.g., Arthur, Bennett, Edens, & Bell, 2003; Kozlowski & Ilgen, 2006; Salas & Cannon-Bowers, 2001). Lastly, researchers have also recommended training strategies for fostering mental-model development (e.g., Marshall, 1995; Mathan & Keodinger, 2005; Smith-Jentsch, Zeisig, Acton, & McPherson, 1998).
Klein and Baxter, however, may be unique and are at least rare in their center-stage placement of the mental model and in the comprehensive way their theory draws together practice, diagnosis, feedback, and learning objectives to guide mental model development. The role that researchers and instructors typically give to training objectives is instead given to the mental model by CTT. In addition, CTT unintuitively describes effective mental models as best attained by periodically rejecting or unlearning the current mental model to adopt a new one. In contrast, most training and most guidance for mental model development (e.g., Kraiger, Ford, & Salas, 1993; Smith-Jentsch et al., 1998) advocates the gradual accumulation, integration, and organization of knowledge, not its rejection or reconstruction.
To the best of our knowledge, no current theory of learning, skill, or expertise acquisition unites teaching practices to focus on continual mental model evaluation and change through an emphasis on metacognitive processes that support sensemaking. With these core teaching practices and primary goal of CTT in mind, in the next section, we provide an overview of sensemaking theory to emphasize the importance of sensemaking to the critical evaluation and development of mental models in learning and expertise acquisition in complex domains.
The Data/Frame Theory of Sensemaking
According to CTT, the process of sensemaking plays a central role in learning, particularly in cognitively demanding domains. Sensemaking can be described as a nearly continuous cognitive activity by which humans understand connections among information, actions, and circumstances in order to detect problems, recognize situations, understand how to achieve goals, predict outcomes, and effectively adapt performance based on those predictions (e.g., Klein et al., 2006a; Klein, Phillips, Rall, & Peluso, 2007). In the context of CTT, sensemaking is the means by which students recognize a conflict between their mental model and the effect of an action or other change in the unfolding situation. It is also necessary for recognizing a mental model inadequacy associated with impaired problem detection, problem assessment, or other sensemaking activities. When a conflict or inadequacy is detected, students use sensemaking to modify their mental models.
According to the Data/Frame Theory of Sensemaking (Klein et al., 2007), the sensemaking process involves retrieving or composing a frame—a knowledge framework analogous to CTT’s mental model—that matches the situation at hand. The frame may be elaborated by integrating new data, questioned to ensure it is appropriate for the situation at hand, and reframed, which consists of replacing an inappropriate frame with one that is a better match to the data prevalent in the unfolding situation. Frames may also be inappropriately preserved by explaining away or distorting conflicting data.
Klein’s conceptualization of sensemaking is drawn from a synthesis of relevant research literature and his own research (e.g., Klein, 1989, 1993, 1998). Especially influential is a 3-yr program of research investigating sensemaking by U.S. Army information operations (IO) specialists (reported in Sieck et al., 2007). In that research program, Klein and his colleagues studied the sensemaking of high- and low-experience (identified as “expert” and “novice”) IO specialists. Data collection consisted of IO specialist assessments of an unfolding situation as understood through a series of situation reports and their detailed accounts of past memorable incidents as well as just-completed periods of work. Iterative development of the model, as described by Seick et al. (2007), was guided by the assessment of specialists’ data, relevant research literature, past relevant studies (e.g., Klein, Calderwood, & Clinton-Cirocco, 1986), and drivers’ retrospective accounts of reorientation attempts after becoming lost.
From these sources, Klein and his colleagues derived nine claims characterizing sensemaking and hypothesized seven forms that sensemaking may take (e.g., question a frame). Their sensemaking claims are supported by the triangulation of patterns they observed across their studies of IO specialists, their prior studies of sensemaking and decision making (e.g., Klein, 1993; Klein, Pliske, Crandall, & Woods, 2005; Klein & Wolf, 1998), and research and theories of others. For some claims, Klein and his colleagues cite a rich body of empirical support (e.g., for the claim sensemaking is used to achieve a functional understanding); for others, the support is more sparse, if only because the claim is challenging to assess—for example, the number of cues from which a frame is inferred (according to Claim 3, the frame is inferred from a few key anchors). Klein and his colleagues describe and explain each claim such that each may be critically evaluated and they present their evidence so that it can be judged in terms of both amount and quality.
For example, Claim 8 says that mental models are typically constructed from stored knowledge fragments and inferences in a “just in time” fashion. In support, Klein et al. describe observing gaps in the frames of IO specialists for most incidents presented to them, and they elaborate by describing one commonly observed gap (ignorance about how cholera is transmitted) in the frame used for evaluating a refugee camp incident. They also discuss related research—in this case, evidence that could be considered disconfirmatory—and its implications for CTT. Specifically, they report a finding by Feltovich, Johnson, Moller, and Swonson (1984, as cited by Klein et al., 2007) suggesting that experts in some domains might have fully developed frames, for example, of the cardiac system. Klein et al. subsequently acknowledge that frames may be more complete in certain domains (e.g., for closed systems such as the cardiac system) but “continue to assert that in most cases even experts are assembling mental models on the spot” (p. 132).
In their discussion of Claim 4, that sensemaking relies on abductive reasoning in addition to logical deduction, Klein et al. cite support from their IO research and similar claims by other theorists, and they provide a verbatim example of abductive reasoning from a study of nursing expertise by Crandall and Getchell-Reiter (1993, as cited by Klein et al., 2007). They again address research that may not completely align with the claim. In this case, they describe a study by Doherty (1993, as cited by Klein et al., 2007) indicating that abductive reasoning may not be an effective sensemaking strategy because of a tendency to generate false correlations. They characterize this effect as an acceptable cost for being able to reason and make decisions when data are limited.
In general, research on sensemaking has been limited in comparison with research investigating, in the information processing tradition, what might be considered microcognitive processes such as perception, attention, working memory, long-term memory, and long-term working memory (LTWM). Together, these focus areas of microcognitive research produce a fragmented picture of sensemaking with, to some degree, a limited utility for understanding cognitive processes in real-world settings (Klein et al., 2003). Although theorists have attempted to tie many of these microcognitive functions together (examples include Anderson’s (1983) Adaptive Control of Thought (ACT) cognitive architecture and the Soar cognitive architecture proposed by Laird, Newell, and Rosenbloom (1987)], they have so far shown little utility in characterizing cognition in natural contexts.
The Data/Frame Theory of Sensemaking and CTT can likewise be contrasted with learning theories rooted in the information processing, or cognitivist, paradigm. CTT and Klein’s sensemaking theory are constructivistic in nature—learning and cognition are described as actively guided by metacognitive processes such as seek and interpret own feedback (according to CTT) and judge frame plausibility and gauge data quality (according to the sensemaking theory). In contrast, the dominant theories of cognition and learning tend to describe passive, automatic processes. These process theories offer connectionist and spreading activation explanations of learning that involve creating, strengthening, and weakening connections and rules [e.g., Anderson’s (1983) ACT and Shiffrin and Schneider’s (1977) theory of automatic and controlled processing]. Ericsson’s (2006) deliberate practice strategy for expertise acquisition makes similar assumptions. Using this strategy, the learner strives to actively seek out challenging tasks and maintain focused effort on improving in order to escape performance ruts and escalate his or her expertise. The learner does not escape those ruts through self-initiated adaptations of knowledge, skill, or learning strategy, however. Instead, deliberate, focused practice at levels of gradually increased difficulty automatically produces the adaptation and associated improvements in skill—that is, the learner does not actively question or shape the knowledge or skill he or she is developing.
Kintsch’s (1988, 1998, 2005) construction-integration (CI) theory of comprehension also describes a passive, bottom-up process of spreading activation, but adds to it a top-down process of calibrating knowledge network activation patterns to preexisting knowledge about what is possible and likely. From the network of calibrated connection strengths, an understanding of the current situation is derived; this understanding is called a situation model. The situation model, as an abstraction of the current situation, is similar to the frames of Klein’s sensemaking theory and the mental models of CTT. It differs, however, in that it is the unavoidable outcome of automatic spreading activation and calibration processes that are driven by the perceived stimuli and the contents and organization of the person’s knowledge base. In contrast, Klein and Baxter argue that sensemaking, and thus learning, should be driven by the learner. They argue that frames and mental models—that is, knowledge abstractions that shape understanding and guide action—can be actively renovated by the sensemaker or learner. We may tend to become invested in our frames and mental models, and may consequently be reticent to make major changes to them (e.g., Feltovich, Spiro, & Coulson, 2001); but doing so is both possible and necessary for adaptive sensemaking and continued learning.
Other theories that give a role to an organizing knowledge structure, such as the mental model that is the focus of CTT, include template theory (e.g., Gobet & Simon, 1996, 1998) and LTWM theory (Ericsson & Kintsch, 1995). These theories propose knowledge structures that can be used to hold large amounts of information active in working memory and LTWM, respectively, and both are consistent, as is CI theory’s situation model, with CTT’s use of a mental model construct (CTT does not make a distinction between WM and LTWM). Where these theories part ways with CTT is, as in the case of the learning theory discussed above, in their tendency to treat knowledge and expertise acquisition as automatic consequences of the cumulative performance of a task or type of work.
Knowledge compilation and mental model derivation can occur passively in CTT and sensemaking theory, too; however, these two theories emphasize the critical role of an additional metacognitive component. Metacognitive processes are processes we use to monitor and manage our cognition, including the more basic perceptual, information storage, and information processing activities as well as the metacognitive processes themselves (e.g., Flavell, 1979). In CTT, they also include recognizing and diagnosing performance problems, managing attention, appreciating the implications of feedback, and unlearning and restructuring mental models (Klein & Baxter, 2009). Metacognition is often not given an explicit role in the above-discussed cognitivist process models, perhaps because their underlying information processing paradigm does not lend itself to representing top-down processes that do not neatly decompose into knowledge elements and connections among them. The importance of metacognition in learning is acknowledged in the education literature (e.g., Hartman & Sternberg, 1993; Sternberg, 1998), but even then, the focus is on knowledge accumulation and integration, which metacognition guides, versus on the use of metacognition to challenge and revamp knowledge.
Research Approach
Although CTT is based on a body of research that crosses the boundaries of multiple complex work domains, little research has been conducted to evaluate or otherwise contribute to the theory; further, its influence may be limited by a lack of implementation guidance and examples. For the reasons we have outlined above that differentiate CTT and the sensemaking process from other theories of learning and skill acquisition, namely, emphasizing the potential of a CTT-based approach in facilitating adaptive expertise acquisition, further research is warranted to identify instructional strategies that align with CTT’s recommended teaching practices. These strategies could aid practitioners in applying CTT and the results of their use might be used to further refine the theory.
This exploratory research employed cognitive task analysis (CTA) methods to study real-world expert instruction in a complex cognitive-work domain to evaluate its consistency with CTT. CTA methods were selected for our approach as they have been found to be essential for identifying and developing effective training recommendations for cognitive work domains (e.g., Crandall, Klein, & Hoffman, 2006; Tofel-Grefhl & Feldon, 2013).
CTA is defined as “the study of cognition in real-world contexts and professional practice at work” and yields rich qualitative data that are able to explicate cognitive work in ways that quantitative data often cannot (Crandall et al., 2006, p. vii). As argued by Klein (1998) and Crandall et al. (2006), research conducted outside the laboratory is useful for gaining insight, improved understanding, and a better foundation from which to develop research hypotheses and models that can be pursued in subsequent studies. Similarly, Pepperberg (2008) argues for the value of observation prior to devising hypotheses as a strategy for identifying more innovative research questions that are grounded in a basic understanding of the variables, their dynamics, and external influences. The use of observation and other qualitative research methods, such as those employed by the present research, may ultimately lead to the generation of meaningfully informed hypotheses suited to empirical testing—at least when compared to hypotheses developed for research without engaging in such preliminary efforts. For the present purposes, CTA methods are used as exploratory means to derive an integrated set of CTT-relevant instructional strategies, which can then be tested empirically in traditional settings.
Further, our approach centered on examining the domain of ATC as it involves a great deal of cognitive complexity (e.g., Durso & Manning, 2008; van Merriënboer et al., 1992) and requires adaptive responses in a wide variety of situations. Specifically, air traffic controllers seek and interpret as many as 27 sources of data, as required by the dynamics and frequencies of the traffic, airspace, communications, and other factors, in order to make sense of a given situation and respond appropriately (see Durso & Manning, 2008). These data can come together and interact to produce an extensive variety of situations for which adaptive expertise is required (Seamster, Redding, Cannon, Ryder, & Purcell, 1993).
Research objectives
The objectives of this research were to assess and characterize the instruction of expert ATC instructors and the corresponding ways ATC novices learn complex cognitive material in order to (a) evaluate the relevance of CTT with regard to this process, (b) propose additions to or refinements of CTT, and (c) identify potential instructional strategies to serve as guidance for the application of CTT and empirical evaluation of CTT-based instruction. We expected that ATC, a demanding cognitive work domain in which adaptive expertise is necessary, would provide the opportunity to find examples of instructional strategies for complex cognitive work. We hypothesized that if evidence was found for the use of strategies consistent with the four core teaching practices of CTT, then this research may offer an indication of real-world instantiations of the theory and our results may serve as preliminary guidance for CTT’s implementation. Further, we expected that those strategies not consistent with the four core teaching practices of CTT, or deviating from the way these strategies are conceptualized by CTT, may offer support for additions to or expansions of the theory. Although we propose our findings as instructional strategies, we acknowledge that future experimental research will need to be conducted in order to establish the effectiveness of these strategies.
Method
Participants
Two professors teaching a visual flight rules (VFR) air traffic control tower (ATCT) course in Embry-Riddle Aeronautical University’s Air Traffic Management program voluntarily participated in this study. An experience questionnaire was given to the professors to elicit further information regarding their experience as controllers and professors of ATC. Results from the experience questionnaire are shown below in Table 1.
Experience Questionnaire Results
Seven undergraduate students (three female), four from Professor 1’s tower course and three from a section of the same course taught by Professor 2, voluntarily participated in this study. This quantity was approximately one third of the total students in each course. The students’ ages ranged from 19 to 21 (M = 20.14, SD = 0.69). The students’ number of years in college ranged from 2.5 to 4 (M = 3, SD = 0.5). The small sample size used in this research is typical of CTA in which large amounts of qualitative data tend to be collected from relatively small numbers of participants (see Crandall et al., 2006, for review; see Chase & Simon, 1973; Gobet & Simon, 1996; Hutchins, Pirolli, & Card, 2007; Kirschenbaum, Trafton, & Pratt, 2007, for examples). In our case, as with many other CTA studies, the small sample size was in part due to limited access to experts, but also a function of the intensive time and resource demands for conducting a CTA (e.g., Militello & Hutton, 1998).
Each participant signed informed consent and audio data collection permission forms. All participants were treated in accordance with the “Ethical Principles of Psychologists and Code of Conduct” of the American Psychological Association. All student participants were compensated $10 for participating in a knowledge elicitation session.
Materials
Two Sony IC Digital Recorders, model ICD-PX312, were used to record course observations and knowledge elicitation sessions. PowerPoint presentations for each of the examined topics were presented on an HP Mini 210-2080NR netbook during discussion of the corresponding topic in both the professor and student knowledge elicitation sessions.
Procedure
Data collection focused on observing and eliciting strategies for teaching and learning the three related flight separation tasks over three instructional phases. More specifically, the complex ATCT tasks same-runway separation, wake turbulence separation, and instrument flight rules (IFR) separation were studied across introductory, practice, and assessment phases of instruction. These tasks were taught sequentially to ATCT students within a 1-month period. Data collection consisted of course observations and knowledge elicitation sessions. Both procedures are described as follows.
Course observations
Both professors’ courses were observed during the in-class introduction to each of the three ATCT separation tasks, subsequent practice sessions, and one performance assessment. Each observed course lasted approximately 45 minutes where the professors would review the details of the separation tasks for approximately 15-30 minutes prior to providing students with a practice session using the ATCT simulators. Therefore, across the two professors, a total of six courses and the subsequent practices session were observed. One performance assessment, for the same runway separation task, was observed for both professors. Course observations were audio recorded and written notes supplemented the recordings. Course observations were used to guide and inform subsequent data collection and interpretation as well as to improve data validity by verifying that participants’ reports were in accordance with observed events (Johnson, 1997).
Professor knowledge elicitation
A single semistructured interview protocol was conducted with each professor to capture each professor’s verbal account of the strategies used to teach each of the three tasks over the 1-month period. Each professor was asked to walk through the month’s instructional activities and to give detailed accounts grounded in specific examples. Artifacts were used to prompt the professors to expand and deepen their accounts; these consisted of course schedules, syllabi, observation notes, and presentation slides. See Appendix A online for the interview protocol that was used for the professor knowledge elicitation. Knowledge elicitation sessions with professors were conducted individually in an office setting and were audio recorded. Interviews with the professors ranged from approximately 45 to 47 minutes and were conducted within 1 month relative to the last course observation and within approximately 2 weeks from the conclusion of the module.
Student knowledge elicitation
A single semistructured interview protocol was conducted with all students to capture their verbal accounts of the instructional activities they experienced during the same 1-month period described by the professors. Artifacts were used as prompts to encourage students to expand and deepen their accounts; these consisted of presentation slides, the course schedule, syllabi, and observation notes. The artifacts were also used to prompt students to recall specific memories of course activities. See Appendix B online for the interview protocol that was used for the student knowledge elicitation. Knowledge elicitation sessions were conducted in a private conference room or an empty classroom and were audio recorded. Interviews with the students ranged from approximately 25 to 45 minutes and were conducted within 1 month relative to the last course observation and within approximately 2 weeks from the conclusion of the module.
Data Analysis
All knowledge elicitation sessions were transcribed from audio to text. Transcripts of knowledge elicitation sessions were broken into data extracts (cf., Braun & Clarke, 2006), where the content of any given data extract was able to stand alone as a meaningful expression, but does not contain more than one idea or concept. Although some approaches to qualitative research advocate a unitization process and assessment of reliability for data extracts independent of code assignment (e.g., Auer-Srnka & Koeszegi, 2007), our approach for assessing reliability of data extracts was inherent to our coding processes detailed below. A total of nine transcriptions (from two professors and seven students) were coded using a coding scheme derived from CTT (Klein & Baxter, 2006, 2009), the Data/Frame Theory of Sensemaking (e.g., Klein et al., 2006a; Klein et al., 2007), and emergent patterns within the data (see Crandall et al., 2006, for details on emergence of patterns in CTA data). Wherein, throughout the coding process, a single concept or idea represented a single data extract as a function of only a single code being assigned to that extract. Thus, the few instances where more than one code could be assigned to a data extract were mutually reassessed by the coders and segmented to fit with the above qualification for a data extract. Therefore, upon completion of the coding process (detailed below), the reliability of the data extracts was κ = 1.00.
Code development
A preliminary set of codes was derived from CTT and the Data/Frame Theory of Sensemaking. These represented an initial set of high-level instructional strategies that might have been identifiable in the data (see Appendix C online for these preliminary codes). An iterative code development process was conducted that involved triangulation among the two researchers’ interpretations of the data and the two relevant theories (see Johnson, 1997, for details on triangulation). To be clear, our process detailed below leveraged a top-down (theory-driven) and bottom-up (data-driven) approach to qualitative data analysis. This type of approach was selected given that the interpretation of qualitative data and the determination of codes is an interactive process that requires openness to modification of theory-driven codes by emergent patterns in the data to mitigate the influence of top-down biases (e.g., Chi, 1997).
For the code development process, the first (A1) and second authors (A2) coded half the data extracts of one professor and one student transcript while simultaneously revising the codes. The code revisions were made to capture patterns within the data that did not map to existing codes as well as recommendations put forth in CTT and the Data/Frame Theory of Sensemaking. When a new pattern reflected an aspect of CTT or the Data/Frame Theory of Sensemaking that was not represented by the codes, the code revision was made to capture both the pattern and the relevant aspect of the theory. Proposed code revisions and example data extracts supporting such revisions were compared, discussed, and agreed upon by A1 and A2. Then, A1 and A2 coded a portion of the second professor and second student transcript. Next, a second iteration of proposed code revisions and coding examples occurred, which led to the development of revised codes (see Table 2 for revised codes). After the development of revised codes, A1 and A2 met weekly throughout the 6-week coding process to compare and discuss the progress of data coding for the remaining data described below. This was done in order to improve descriptive validity through critique of one another’s data interpretations (Johnson, 1997).
Coding Frequencies, Percentages, and Number of Participants Represented
The code “develop fluency in use of mental model” appears twice in this table because it supports two of the CTT teaching practice categories. Therefore, percent data and quantity of participants represented by this code represent the same set of data extracts in both cases.
Changes from the preliminary set of codes to the revised codes primarily took the form of modifying the tense and associated qualifiers that allowed for a cross-mapping of codes across student and professor transcripts. That is, many of the initial instructional strategies detailed in the preliminary set of codes are still evident in the revised codes, but there is greater nuance and specificity in the revised codes characterized by both theory and the data.
Data coding
Data coding was conducted by one of the code development researchers (A1) and a third researcher who was the fifth author of this paper (A5). A5 was trained for the coding process using a similar method to that used for code revision. First, to improve theoretical validity in terms of matching patterns in the data to theory, A5, who had no involvement in the interviews, was familiarized with CTT, the Data/Frame Theory of Sensemaking, and the ATC course artifacts. Further, A5 was trained to assign the codes to data extracts utilizing a subset of the transcriptions. For this subset, which consisted of the transcriptions of one professor and one student, A5 coded data extracts in one-page increments and compared and discussed each increment with A1. After A5 went through the transcription subset, the two coders worked independently.
Interrater reliability was first analyzed for the set of initial independent coding. Based on the initial independent coding reliability, codes used dissimilarly were discussed by the coders and then assigned a reconciled code. This led to a set of reconciled coding that was also subjected to interrater reliability analysis. After the data were coded and assessed, the findings and conclusions were evaluated and interpretations were critiqued by both A1 and A2 to improve descriptive and interpretive validity. Participant feedback was solicited from the two ATC professors by presenting them both with the proposed findings and conclusions and requesting their critical feedback. Moreover, interpretive and theoretical validity were improved through peer review in the form of a comprehensive presentation of the literature review, methods, results, and conclusions to a committee of external researchers. With the feedback of participants, the external committee, and recommendations from reviewers, the researchers engaged in reflexivity, a critical self-reflection on any biases the researchers may have had, in attempt to disconfirm their interpretations and alignment of the data with CTT as opposed to other theories of learning and skill acquisition (see Johnson, 1997, for details on the strategies used here for improving validity in qualitative research). The outcomes of this reflexivity process are detailed in a subsection of our “Results and Discussion” section titled “Support for CTT.”
Results and Discussion
Although full findings from this research can be found in Wiltshire (2012), the results presented here consist of patterns found in the coded interview data that are supported by course observations and artifacts that are in accord with the specified research objectives. A total of 627 data extracts were coded. The number of data extracts obtained from each participant was as follows: P1 = 106, P2 = 106, S1 = 48, S2 = 65, S3 = 56, S4 = 64, S5 = 76, S6 = 45, and S7 = 61. However, the data reported herein account for 89.3% of all coded data. The remaining 10.7% of the data are not elaborated upon in this manuscript because they were unrelated to learning and instruction and were primarily coded as background material (see Table 2 for details of coding frequencies, percentages, and number of participants represented by a given code).
As noted previously, this exploratory research aims to provide insight into the instructional strategies utilized by expert instructors in a complex cognitive work domain, and as such, the findings reported here are meant to serve as preliminary strategies to guide the transition of CTT from theory to practice. Taken together, the results presented herein extend beyond the four core teaching practices of CTT by providing a refined level of specificity with regard to how CTT might be applied as a function of the proposed lower level instructional strategies mapping to the core teaching practices and also mental model development. However, it should be noted that these instructional strategies should first be applied and evaluated in experimental settings prior to their use in real training and educational settings. As such, this integrated “Results and Discussion” section addresses the objectives of this research and is structured as follows. Findings and implications related to general mental model development are presented prior to findings and implications for each of CTT’s four teaching practices (e.g., diagnosis, learning objectives, practice, and feedback). First, however, results of the interrater reliability analysis will be detailed.
Interrater Reliability Analysis
The percentage of direct agreement for initial independent coding of the data was 57% and Cohen’s kappa coefficient was .51 (range across transcripts: κ = .35 to .66). Based on criteria set forth by Banerjee, Capozzoli, McSweeney, and Sinha (1999), κ = .51 represents a fair level of agreement beyond that due to chance. The percentage of direct agreement for the reconciled coding was 93% and Cohen’s kappa coefficient was .90 (range across transcripts: κ = .83 to .97). Based on the criteria of Banerjee et al. (1999), the reconciled coding had a substantial level of agreement beyond that due to chance.
Mental Model Development
In CTT, the continual formation of increasingly accurate mental models is essential for learning in a cognitive work domain—an emphasis absent from most cognitive theories of learning. Almost half of the data extracts described learning or instruction in terms that related to mental model development. These data, which accounted for 45.9% of all extracts, fell into three subcategories. The code “Teach/learn an element of mental model” accounted for 6% of all coded data and was assigned to data from eight participants. The code “Form rudimentary mental model” accounted for 32.1% of all coded data, was assigned to data from all nine participants, and was the most prevalent of all codes. The code “Develop fluency in use of mental model” accounted for 7.8% of all coded data and was assigned to data from seven participants. Table 3 provides a description of each code associated with mental model development as well as example data extracts.
Mental Model Development Codes, Descriptions, and Data Extracts
Consistent with CTT, these results indicate that mental model development is likely fostered within this course, thus supporting the notion that the continual formation of increasingly accurate mental models is essential for learning in a cognitive work domain. This is supported by the applicability of the three mental model development codes detailed above. Of particular interest is the relationship between the rather limited assignment of the “Teach/learn elements of a mental model” code with the most frequently assigned code “Form rudimentary mental model.”
The contrast in assignment of the former two mental model codes was likely due to the fact that, given this was an introductory course to ATCT, students did not have much of a mental model to begin with. In addressing this need, the instructors seemed to help the students to form rudimentary mental models through the integrated way they presented the course material to students. Elaborating on this notion, Chi (2008) posits that the assignment of concepts to categories is an essential learning strategy that benefits mental model formation. Early in the course, students were taught aircraft types, and as they moved into the complex separation tasks, aircraft types were assigned classes and categories depending on which separation rule was being taught. These classes and categories differed with respect to the amount of distance and time required to separate different aircraft types at different runway locations. This is one strategy that potentially helped to facilitate the formation of rudimentary mental models.
In addition to the classification and categorization strategy for learning the material, instructors also primarily presented the material in ways that facilitated its integration with other material. Specifically, material in this course was mainly presented in formats that integrated different forms of knowledge, which is likely to further support the students’ formation of rudimentary mental models. The course instructors required students to self-study the material, primarily through online PowerPoint presentations, prior to learning the material during class time. The self-study components were presented in such a way as to represent and help facilitate the integration of multiple forms of knowledge. For example, students were presented with not only the required rules and regulations but also visual examples of aircraft types organized into their categories and classes, airport diagrams with directional and locational cues for moving aircraft, and scenario questions that provided context for moving various combinations of aircraft from varying runway locations.
The practice of supporting mental model development is exemplified by this integrated and diagrammatic presentation of material used in this course. Figure 1 shows an example presentation slide that integrates text and diagram to convey temporal and directional cues as opposed to just factual text. This slide represents part of a prerecorded lecture that is embedded within the PowerPoint presentations, which students self-study prior to class. Although the data collected in this research are not suitable to establish a causal link between this instructional strategy and effective mental model development, a growing body of experimental research is providing such evidence. For example, experimental findings have shown that participants in training conditions that include diagrammatic presentations are able to better interconnect information and form more robust knowledge structures (Fiore, Cuevas, & Oser, 2003) as well as perform better on complex tasks (Lewandosky, Dunn, Kirsner, & Randall, 1997).
PowerPoint presentation slide used in self-study portion of the ATCT course.
In sum, we can characterize the mental model codes assigned to the data as high-level instructional strategies that supported mental model development. Accordingly, Table 4 presents these instructional strategies and then gives descriptions of ways each strategy was and can be implemented. We expect that these strategies, abstracted from the data and presented throughout the following subsections, are at a general enough level that would be applicable to a number of other instructional contexts, though future research will need to empirically evaluate this claim.
Instructional Strategies Used That Could Contribute to Mental Model Development
Diagnostic Assessments to Identify Flaws in Mental Models
In CTT and other research on the acquisition of knowledge, a diagnostic component that allows for the opportunity to identify flaws in mental models is essential (e.g., Chi, 2008; Ericsson, Krampe, & Tesch-Romer, 1993; Feltovich et al., 1993; Klein & Baxter, 2006, 2009). Over 10% of all data extracts referred to diagnostic strategies that related to the identification of mental model weaknesses and inaccuracies. These extracts fell into two subcategories. The code “Reveal/recognize weakness in mental model” was the second most used code. It accounted for 9.8% of all coded data and was assigned to data from all nine participants. The code “Anticipate weakness in mental model” accounted for 2.9% of all coded data and was only assigned to extracts in the professors’ transcriptions. Table 5 provides a description of each code associated with diagnosis as well as example data extracts.
Diagnosis Codes, Descriptions, and Data Extracts
From the course artifacts, observations, and knowledge elicitation data, six specific types of diagnostic assessments were identified that likely were used to help students to identify flaws in their mental models. Three assessment types helped students identify flaws on their own and included online scenario questions, online quizzes, and online self-assessments. To illustrate the notion that students actively used these assessment opportunities to identify their weaknesses, S5 stated the following: “When I took the quizzes, I always did it without notes the first couple of times and if I was struggling, then I might go and look at the notes.” Three other assessment types were used that likely helped both the professors and students identify mental models’ flaws. These included in-class scenario questions, block tests, and performance verifications. The following quote by S7 describes how the professors were described as able to identify weaknesses during the performance verifications: “Once we had our performance verifications. … [The professor] now knows how everyone is doing. Then, the next time we would come to class, he could address what most people had a problem with; how to correct it and what to do from now on.”
In addition to the explicit assessment types, the data suggest that an opportunity for mental model weaknesses to reveal themselves was during the simulation component of the class. For example, P2 stated: “Then they are going to apply it [in simulation scenarios] and we see that they really didn’t understand it.” Relatedly, Klein and Baxter (2009) posit that virtual environments, such as the ATCT simulator used in this course, provide students with the opportunity to see how their actions play out, thus allowing for flaws in mental models to be revealed.
From course observations and patterns in the data, working with teammates seemed to provide an opportunity for the students to recognize and reveal weaknesses in their teammates’ mental models. For example, S4 stated, “The whole point of working together is that you can catch other peoples’ mistakes.” In this course, students were encouraged to work together to help out their teammates and thus reported team interaction as beneficial. Accordingly, another implication for the practice of diagnosis is that, in this course, even when unsupervised by the professor and lab assistants, students had the opportunity to mutually diagnose and identify flaws in each others’ mental models.
Diagnostic strategies, such as those identified in this course, serve the purpose of identifying flaws in students’ mental models in such a way that ideally, as Redding (1989) suggested, over time students’ mental models more closely approximate that of an expert. In sum, we can characterize the codes assigned to the data as high-level instructional strategies that supported diagnosis. Accordingly, Table 6 presents these instructional strategies and then gives specific descriptions of ways each strategy was implemented and thus can be in a broader instructional context.
Instructional Strategies Used That Could Contribute to Diagnosis
Learning Objectives That Emphasize Sensemaking and Encourage Reflection
According to CTT, a critical learning activity that should be included as a learning objective is an emphasis on sensemaking and the reflection on new information in order to integrate this new information with existing mental models. The encouragement and development of student sensemaking and reflection in the ATCT course was suggested by 4.6% of all data extracts. These extracts fell into two subcategories. The code “Metacognitive: self-reflection, self-evaluation” accounted for 3.3% of coded data and was assigned to data extracts from seven participants. The code “Weave new learning into existing knowledge; connect new information to existing knowledge” accounted for 1.3% of the coded data and was assigned to data from five participants. Table 7 provides a description of each code associated with the CTT learning objectives component as well as example data extracts.
Learning Objective Codes, Descriptions, and Data Extracts
Consistent with CTT, course observations and coded data suggest that students participated in opportunities for reflection on new information and the information’s’ interrelationships with prior knowledge. According to Klein and Baxter (2006, 2009), these activities are in support of the deliberate and continual restructuring of students’ mental models. However, though there was evidence of these instructional and learning strategies, in its current form, CTT is not clear about the role of learning objectives. Klein and Baxter (2006) state that “[novices’] learning objective is to employ sensemaking to generate initial mental models of cause/effect stories…” (p. 5). But what is not clear is how this notion relates to traditional learning objectives. For example, in CTT are traditional learning objectives to be eschewed in favor of a single sensemaking objective as described above? Or are sensemaking learning objectives meant to be complementary to traditional learning objectives? Given these questions, among others, we aim to review our findings in order to detail the refinements and implications that may help to provide some clarity on this issue. In particular, the course instructors relied on traditional knowledge- and performance-based learning objectives for their classes. Despite the objectives focusing on knowledge- and performance-based objectives, the goals of sensemaking and reflection seemed to exist throughout the course activities, albeit in an implicit form.
For example, the code “Metacognitive: self-reflection, self-evaluation” represented the incorporation of reflection during learning in this course. Further, there was evidence that in addition to the many diagnostic activities that could engage reflective mechanisms, the professors also relied on a prompting technique that encouraged metacognitive activities before, during, and after observations of ATCT simulation practice. Specifically, at various phases of practice, the professors were observed to question the students regarding aspects of their performance in order to get the students to assess the degree to which they really knew and understood that they were applying the separation rules appropriately.
An additional code “Weave new learning into existing knowledge; connect new information to existing knowledge” represented a learning activity that was an implicit goal of the course, related to the recommendation of CTT, and was indicative of reflection on new learning. For example, S3 stated, “The first thing we learned was same-runway separation and then they added wake turbulence. We learned that and it’s like a refinement, [where] you’ve added another level of sophistication to the rules.” In addition to references to this activity in participant transcriptions, during each of the three observed class periods, professors recounted material from the preceding class and related it to the topic of the day by describing various scenarios and thus seemed to facilitate the connection of new information with existing knowledge.
Therefore, based on observations of the course and evidence in the knowledge elicitation data, sensemaking seemed to be encouraged in this course and the recommendations of CTT are indeed reflected in the instructional strategies used. Though there were no explicit learning objectives targeting student sensemaking or reflection, the incorporation of reflection on performance with simulation scenarios combined with instances where students were prompted to reflect and integrate new information with preexisting knowledge suggests that sensemaking and reflection were useful. However, as a refinement to CTT, we submit the following:
Refinement 1: The emphasis in CTT on reflection may not encompass all the cognitive learning activities that contribute to the development of increasingly accurate mental models; instead, we propose an additional emphasis on developing self-regulation skill.
In particular, we make this claim given that the term reflection implies that this process occurs after a given instructional event. This assertion is consistent with Fiore and Vogel-Walcutt’s (2010) proposition that metacognitive prompts, such as those used during observations of ATCT simulation practice, can facilitate self-regulation. Specifically, self-regulation is defined as “the ability to monitor and modulate cognition, emotion, and behavior, to accomplish one’s goal and/or to adapt to the cognitive and social demands of specific situations” (Berger, Kofman, Livneh, & Henik, 2007, p. 257), which is similar to the definition of sensemaking detailed previously. It has long been thought that there are inherent benefits of metacognitive and self-regulatory processes (e.g., Flavell, 1979), and indeed, recent meta-analyses have shown that self-regulated learning strategies, and in particular, those that would be considered metacognitive, contribute to better learning outcomes in both a classroom and workplace environment (Dignath & Bütner, 2008; Sitzmann & Ely, 2011). Given the surrounding research supporting the benefits of self-regulation on learning, we argue that this emphasis, while seemingly a trivial matter of terminology, may provide further lines of inquiry given the greater conceptual breadth encompassed by self-regulation as opposed to just reflection.
With regard to the present purposes, we suggest that metacognitive prompting as an easily implementable instructional strategy shows promise given that Fiore and Vogel-Walcutt (2010) assert that this technique can systematically prompt a learner to assess their learning performance during preparation, execution, and reflection phases of a given instructional event. An example of a metacognitive prompt supporting preparation would be if, prior to practice, the professor asked which taxiway was indicative of the necessary distance required for separation; whereas, asking students while they are practicing if they have the necessary separation is an example of a prompt supporting execution. Finally, asking the students after a practice session questions regarding how they did and what they could do better are examples of prompts that support reflection. Therefore, an implication for the practice of encouraging sensemaking may be including metacognitive prompting that supports preparation and execution, in addition to reflection. Further, the incorporation of metacognitive prompting, as a component of CTT, may serve to engage sensemaking processes as an explicit instructional strategy and may help novice students to overcome the sensemaking paradox by facilitating the development of appropriate metacognitive skills (cf., Butcher & Sumner, 2011).
In CTT, Klein and Baxter (2006, 2009) assert that additional learning objectives that emphasize sensemaking are required, above and beyond traditional learning objectives, which emphasize sensemaking. Recall that we found the role of learning objectives unclear in CTT. In terms of the implications these findings hold for CTT regarding the role of learning objectives, we posit that the instructional strategies that encouraged sensemaking in this course, but were not specifically written as formal objectives, may be as central to student learning as the goals included in formal objectives. Examples of the types of instructional strategies we are suggesting here can be found in Tables 6, 8, 10, and 12 (further empirical work is needed to assess the type and degree to which each of these engages sensemaking processes, as well as what others may do so). Therefore, in light of our findings and the above assertion, the following refinement to CTT is proposed:
Instructional Strategies Used That Could Contribute to Encouraging Sensemaking
Refinement 2: Instructional strategies that systematically engage student’s self-regulatory and metacognitive processes may be all that is necessary to accomplish the objective of sensemaking as opposed to the addition of knowledge- or performance-based learning objectives and/or course content.
In sum, we can characterize the codes assigned to the data as high-level instructional strategies that supported learning objectives. Accordingly, Table 8 presents these instructional strategies and then gives descriptions of ways each strategy was implemented and thus can be in a broader instructional context.
Practice That Incorporates Sensemaking
According to CTT, students should be given ample opportunities to engage in practice that incorporates sensemaking. Approximately 14.4% of the data extracts were related to practice of this type. These extracts fell into three subcategories. One subcategory was also considered a subcategory of the mental model development component (i.e., “Develop fluency in use of mental model”). The code “Emphasis on performing or applying knowledge” accounted for 5.1% of all coded data and was assigned to data from seven participants. The code “Develop fluency in use of mental model” accounted for 7.8% of all coded data and was also assigned to data from seven participants. The code “Assist/improve the directing and shifting of attention” accounted for 3.5% of all coded data and was assigned to data from five participants. Table 9 provides a description of each code associated with practice as well as example data extracts (see Table 3 for a review of “Develop fluency in use of mental model”).
Practice Codes, Descriptions, and Data Extracts
Consistent with CTT, these results suggest that practice may be essential for helping students gain proficiency and engage in sensemaking within this domain. The three codes detailed above represented three general goals captured by codes that all seemed to emphasize a practice component that was related to use of the ATCT simulation. Broadly, the data suggest that the course was largely based on performance, that as students practiced certain aspects of performance required less effort, and that there was a need to manage student attention as advocated by CTT. Observational data revealed an additional compare-and-contrast case instructional strategy used by the professors prior to engaging in simulation-based practice. Below, we review the findings from this study regarding practice and present these as specific instructional strategies.
The majority of participants emphasized that being able to perform or practice using simulation scenarios was essential to learning in the ATCT course. For example, P2 stated, “It is the application [of knowledge] though, I am convinced, the way we have it [the course] set up. The application is really where it drives it home as to what the concept is that we are trying to teach them.” Similarly, students indicated that being able to practice with simulation scenarios helped them learn. For example, S2 stated, “In this class, it makes you have to actually learn it because you have to use it.” Another student, S7, stated, “I would memorize it and because we practice so often, and I used that, I could say I kept that knowledge fairly well.”
In addition to the reported utility of performing and applying knowledge using the ATCT simulation, the data suggested that as students practiced, the amount of cognitive effort required for a given performance was reduced. In particular, the context in which the code “Develop fluency in use of mental model” was assigned suggests this was done in two ways: recognition of stimuli (e.g., aircraft types) and recognition of patterns (e.g., combinations of aircraft types with their locations on the runway and the associated separation requirements). The following quotes describe the process in which recognition of stimuli becomes more efficient. P2 said, “You learn it [aircraft types] to where you don’t have to think too much about it,” and S3 stated, “You basically learn your types of aircraft and when you see that aircraft you automatically think ‘that’s a heavy.’ The way you learn it is just practicing it.” The following quote from S2 describes the way practice helps recognition of patterns become more efficient: “You get used to seeing planes at an intersection that are going to wait 3 minutes if they are this size. You just kind of come to recognize, ‘ok he is this size and he is at this intersection, 3 minutes.’ So you can just kind of look at it.” From the course observations and patterns within the data, it seemed that as students gained more experience through practice, they were able to more easily recognize patterns of aircraft and the associated separation rules such that performance required less cognitive effort.
Consistent with recommendations of CTT, professors and lab assistants seemed to help students manage their attention during practice such that meaningful cues were recognized and cause-effect relations were noticed. This likely allowed students to form stronger causal relationships. For example, S2 stated, “If you don’t see two planes hitting, you are not going to know they are hitting unless someone points it out to you or draws your attention to it.” Similarly, P2 stated, “They start seeing their lab assistants and their professors pointing out to them that aircraft should be lining up to be out there on the runway. You should be clearing him for take-off already. You can clear him to land because you have this separation.”
In addition, the professors were observed to use a compare-and-contrast case instructional strategy that was similar to the recommendations of Feltovich et al. (1993), Fowlkes, Norman, Schatz, and Stagl (2009), and Schwartz and Bransford (1998). That is, prior to practice using the ATCT simulation, students compared and contrasted cases in the form of scenario questions and simulation scenarios with other scenarios featuring different separation rules, which the literature describes such strategies as contributing to the development of deeper and more flexible knowledge structures. The scenarios or cases were coupled with lectures during which professors pointed out what was important in the scenarios.
Overall, the data were relevant to CTT in that practice was emphasized and strategies that engaged the students in sensemaking that likely supported mental model development were employed. Importantly, the practice strategies utilized in this course share similarities with how Simon and Chase (1973) describe the development of expertise in chess. That is, expertise in chess is developed through practice in which an individual builds up a vast repertoire of patterns in long-term memory such that patterns become easily recognizable and performance becomes more efficient. In sum, we can characterize the codes assigned to the data as high-level instructional strategies that supported practice. Accordingly, Table 10 presents these instructional strategies and then gives descriptions of ways each strategy was implemented and thus can be in a broader instructional context.
Instructional Strategies Used That Could Contribute to Sensemaking Practice
Feedback That Indicates How Performance Can Be Improved and Prompts Sensemaking
According to CTT, feedback should inform students of how performance can be improved and encourage students to seek and interpret feedback on their own. Approximately 20.6% of all coded data directly pertained to feedback. These extracts fell into one of seven subcategories; however, only the two most frequently assigned subcategories are reported here. The code “Give/receive process feedback” accounted for 6.8% of all coded data, was the most prevalent of the feedback codes, and was assigned to data from all nine participants. The code “Supplement inadequate mental model; seek or provide information about what student should be doing” accounted for 6.5% of all coded data, was the second most prevalent feedback code, and was assigned to data from all nine participants. Table 11 provides descriptions of the two feedback codes as well as example data extracts.
Feedback Codes, Descriptions, and Data Extracts
Consistent with CTT, process feedback was the most prevalent type of feedback provided, which likely informed students of how to improve their performance. When comparing the extensive assignment of the code “Give/receive process feedback” to the characterization of a single data extract by “Give/receive outcome feedback,” findings supported the notion that, for cognitive work, it may be more important to know how performance can be improved (i.e., process feedback) than to just know that the performance was wrong (i.e., outcome feedback). These data suggest implications for instructional strategies that can be used to support feedback and refine CTT.
Although CTT advocates the use of process feedback, it also proposes that limits be placed on the extent to which feedback is given by external sources. Klein and Baxter (2006, 2009) assert that this limitation is needed so that students learn to seek and interpret feedback on their own, a capability that will allow them to continue learning and improving long after they complete their formal training. In comparison, professors and lab assistants provided ATCT students with robust external feedback that gradually decreased as the course progressed and student performance improved. This may be a useful technique for encouraging students to ultimately seek and interpret feedback on their own. For example, P1 stated, “As the days go by, our input diminishes to the point where at the end of the semester, theoretically, we shouldn’t be saying anything to the students; we are just watching them run the airplanes. I mean they should be applying all those little inputs that we gave along the way. You know, giving them feedback as they went.” A refinement for CTT, suggested by this finding is:
Refinement 3: Students provided with robust initial process feedback, which is decreased as they advance in the course and engage in sensemaking, may develop the capability to seek and interpret feedback on their own provided continuous sources of feedback are available.
Further qualification is needed here. Given the robust nature of feedback available to students as a function of the diversity of feedback types and sources, feedback can be thought of as almost continuously available (cf., Hoffman et al., 2014). Indeed, Bransford and Schwartz (2009) note that when feedback is continuously available, students are likely to engage in self-assessments. Therefore, providing students with high levels of process feedback during early stages of learning, decreasing the rate at which it is provided, while supplementing this with “continuously available” sources of feedback that students can seek on their own, may ultimately contribute to more advanced levels of proficiency. At least this seems to be the case in the context of the current study.
In addition to the strategy of providing process feedback, the code “Supplement inadequate mental model; seek or provide information about what student should be doing” accounted primarily for a strategy in which students recognized a weakness in their mental model and then sought the information from their teammates, lab assistants, professor, or memory aids. The frequency of this form of support within the data is consistent with the postulate of CTT that as novel events unfold, students are able to construct more accurate mental models by seeking the information on a just-in-time basis. Moreover, this just-in-time source of feedback may strengthen the perceived relationships between causes and effects and thus serves to enrich students’ mental models (e.g., Klein & Baxter, 2006, 2009). In short, students in the class were provided robust initial process feedback that decreased as the course progressed and students demonstrated in simulation scenarios that they knew the material; however, when students were presented novel scenarios, they were observed to recognize their limitations and seek the feedback or information necessary to achieve fluid performance.
Notably, the data indicated that one source of just-in-time information, regarded as beneficial by both students and professors, was other students. The professors strove to create a cooperative and team-oriented environment that approximated teamwork in real-world ATC operations. P1 said, “You want them to be able to talk back and forth between each other and point out maybe where someone didn’t do something quite properly or correct without the feeling of being slighted.” Similarly, S3 stated, “What helps me learn the best or what has helped me? Being able to talk with my classmates about what problem we are working on. Hearing them say ‘Oh you need to do this’ and I’d be able to say ‘why’ and they would explain it; telling them ‘hey you need to do this’ and then explaining it to them.” Though not emphasized in CTT, this finding points to the benefits of adopting the instructional strategy of fostering an environment that encourages students to engage in joint-sensemaking processes with each other (cf., Klein, Wiggins, & Dominguez, 2010).
To summarize, as students recognized weaknesses in their mental model by means of diagnostic strategies, engaging in sensemaking, practice in simulation scenarios, and receiving and providing feedback, they drew upon feedback resources consisting of teammates, lab assistants, professors, and memory aids. This seemed to help ensure that students were able to understand causal relationships as they occurred. The feedback resources in this course seemed to allow for students to form and revise their mental models through recognition of cause-and-effect relationships, flaws in their mental models, and strategies to improve performance. Students have the opportunity to see their actions play out in the ATCT simulator, and when they are unsure of the proper action to take, there are numerous sources of information they are able to use to supplement their mental models and continue performing. In sum, we can characterize the codes assigned to the data as high-level instructional strategies that supported feedback. Accordingly, Table 12 presents these instructional strategies and then gives descriptions of ways each strategy was implemented and thus can be in a broader instructional context.
Instructional Strategies Used That Could Contribute to Process Feedback
Support for CTT
This study involved comparing CTT principles with real-world instructional practices and learning experiences in a course taught by experts whose effectiveness is critical in their high stakes domain of ATC. Although we found evidence of all CTT-advocated teaching and learning practices in this course, it is possible to argue that our evidence is equally supportive of other learning theories. We revisit here our evidence and its ability to distinguish CTT from other theories. In particular, we wish to contrast CTT with theories that emphasize the incremental and passive development of knowledge and skill, such as CI theory, LTWM theory, template theory, the Soar and ACT architectures, claims underlying the deliberate practice strategy, and automatic and controlled processing theory. CTT, on the other hand, emphasizes active metacognition to support sensemaking about what is being learned.
A number of our findings do not clearly support CTT over other theories. This may be partly a consequence of our coding scheme, which did not, for example, distinguish between the passive receipt and active seeking of feedback or shed light on how students used practice opportunities. The raw data contributing to some of these findings, however, offer hints of support for active learning and sensemaking. For example, data extracts representing the two feedback codes suggest that students at least sometimes actively sought feedback: S2: I really think it was just practice [that helped me learn]. A lot of practice, and asking questions, mainly, you know a couple times you would double check with your partner, “Hey this is 3 minutes right?” Then you just ask it enough and you are like “Ok, it’s 3 minutes.” S7: He would be walking around and if he was nearby, I would grab him real quick and be like, “Professor I don’t understand this, can you explain it?” and he would.
Some of our other findings more clearly support active learning and sensemaking by the students in addition to encouragement of active learning and sensemaking by the instructors. These findings favor CTT over other learning theories. Specifically, support was found for the following data categorization codes:
form rudimentary mental model (mapped to 32.1% of data extracts),
reveal/recognize weakness in mental model (mapped to 9.8% of data extracts),
anticipate weakness in mental model (mapped to 2.9% of data extracts), and
assist/improve the directing and shifting of attention (mapped to 3.5% of data extracts).
Data extracts that fell within these categories convey a learning process that involves a continuous sensemaking effort, a process captured by this quote: S1: It allowed me to get a handle on the separation requirements before seeing it in the simulator. I kind of knew how to work the stuff and how to organize it rather than just reading the size aircraft and the times and things. We were able to think through it and be a little bit better prepared so when we saw it, it wasn’t self-explanatory but it was much easier to understand.
Our results are unable to rule out relatively passive knowledge and skill development; however, they are able to support a role for metacognition, active mental model development, and the idea that mental models should be continually checked for weaknesses and misconceptions. Although the data do not present clear examples of mental models being completely renovated after a misconception is found, they indicate that the instructors and students were open to and, in fact, worked toward discovering whether students really understood the concepts they were trying to learn. It would be interesting to contrast this finding with instruction and learning in an advanced class in which participants may be more likely to assume that students’ base mental models are accurate and immutable.
Limitations
Although this research specifically focused on CTT, it is limited in the extent to which it can definitively rule out other theories of learning, as alluded to in the section above. An exhaustive coding process (i.e., the development of a coding scheme for each theory of learning or skill acquisition and its application to the data) would likely be needed to do so; however, this would have been beyond the current scope of our study. Instead, an integrated set of instructional strategies were proposed that map to the centrality of mental model development and four teaching practices of CTT. In moving forward from this research, further efforts are required to assess what integrated sets of instructional strategies may be associated with other theories of learning. In doing so, experiments can be conducted that ultimately assess the instructional effectiveness obtained as a function of a coherent set of instructional strategies grounded in and mapped to various learning theories.
This research also does not claim to have determined how best to implement CTT or otherwise instruct to produce adaptive expertise. Limitations of its methods prevent it from doing so. In particular, we were not able to obtain final course assessments for interviewed students, track students’ future progression, or compare the learning outcomes of courses taught in ways compatible with versus in conflict with CTT. In addition, this study relied on verbal, self-report data obtained using knowledge elicitation methods. The accuracy of self-report data, no matter how obtained, will tend to be degraded because knowledge is not always stored in an easily verbalized format and because it is subject to the fading and imperfections of memory—an issue prevalent even in experts (e.g., Tofel-Grehl & Feldon, 2013). A number of steps were taken, however, to improve the accuracy of our self-report data, as described in the “Method” section, including the complementary observation of classes and the use of recorded class events and course artifacts to cue interviewee memories and facilitate recall. Nevertheless, it is still possible that important aspects of teaching and learning activities were not captured at all or were not captured in a complete or completely accurate way.
Conclusion
To reiterate, the objectives of this research were to (a) evaluate the relevance of CTT for the process of instruction by expert ATC instructors and the corresponding ways ATC novices learn complex cognitive material, (b) propose additions to or refinements of CTT, and (c) identify potential instructional strategies to serve as guidance for the application of CTT and empirical evaluation of CTT-based instruction. Accordingly, the “Results and Discussion” detailed in the previous section addressed each of these objectives demonstrating support for the core teaching practices of CTT, three explicit refinements to the theory, and an integrated set of 15 instructional strategies that can serve as guidance for the application of CTT and empirical evaluation of CTT-based instruction. In sum, this exploratory research was instrumental for gaining insight into the education of novices in a complex cognitive work domain. These strategies may be useful for helping students to overcome the sensemaking paradox (i.e., a barrier to learning where novices may lack both the required domain knowledge and metacognitive skills for rich comprehension and integration of relevant material into their existing mental models; Butcher & Sumner, 2011) as well as for facilitating the acquisition of adaptive expertise and accordingly warrant further study. Though this research cannot claim to have determined how best to facilitate the acquisition of adaptive expertise for cognitive work, it serves as a starting point, an initial framework, and set of instructional strategies and teaching practices on which to build, so that the recommendations of CTT can be empirically evaluated and compared with traditional training methods and in turn applied to training and educational systems.
Ideally, empirical evaluation of the applications of CTT detailed in this research would demonstrate the potential to foster learning and continual development long after structured training and education commence. That is, the sensemaking teaching practices reviewed in this research and the proposed integrated set of instructional strategies may prove foundational for providing novices with the capacity to continually refine and attune their knowledge such that the mental models that guide their interactions within the domain of instruction steadily become more enriched, accurate, and attuned to the demands and nuances of that domain. What is important to emphasize with regard to CTT is that at any level of learning, whether it is for professional work in complex cognitive domains or within the traditional education system, students need to learn how to learn on their own, and in order to do so, they must have a framework with which to determine what information to seek, when to seek that information, and when the information is irrelevant. That is, what is necessary is a guiding framework, such as the teaching practices outlined in the present research, that can provide students and professionals with both foundational domain knowledge and the metacognitive skills necessary to overcome the sensemaking paradox (see Butcher & Sumner, 2011) and, in turn, progress toward a path of adaptive expertise.
In conclusion, this research documented the employment of strategies that were relevant to many of the recommendations of CTT. Their employment by instructors who had no awareness of CTT or sensemaking theory, but who had to succeed in preparing students for complex cognitive work, is in itself testimony for the value of CTT and the need for its further study and application in training and educational systems. More broadly, both instructors and students may benefit from the strategies identified in this research; however, further research is warranted to first evaluate their combined effectiveness. That is, although the sensemaking teaching practices can be implemented using strategies identified in this research, we encourage the controlled study of the strategies. In turn, we expect the evidence will show that adopting such an approach may be pivotal in preparing individuals for success in many of today’s complex cognitive work domains.
Footnotes
Acknowledgements
The authors wish to thank all those who helped to make this work possible. This includes Elizabeth Blickensderfer and Jennifer Fowlkes for their thoughtful critiques of our data interpretations, all the students who volunteered to participate in this study, and last but not least, the three reviewers who provided very helpful and constructive feedback that greatly improved this manuscript.
Travis J. Wiltshire is currently pursuing his PhD in Modeling & Simulation at the University of Central Florida (UCF) and works as a Graduate Research Associate in the Cognitive Sciences Laboratory at the Institute for Simulation & Training (IST). He completed his MS in Human Factors & Systems at Embry-Riddle Aeronautical University and previously worked as an instructional designer.
Kelly J. Neville, PhD, is an Associate Professor of Human Factors and Systems at Embry-Riddle Aeronautical University in Daytona Beach, FL. She has worked in the field of human factors for 22 years, studying team coordination; expertise in fast-paced, high-stakes domains; training requirements for complex work; and stress effects on cognitive performance. She earned an MA and PhD in Experimental Psychology from Rice University.
Martin R. Lauth is an Associate Professor of Applied Aviation Sciences at Embry-Riddle Aeronautical University. He has a wide range of applied research interests and enjoys cross-disciplinary collaboration. Prior to coming to Embry-Riddle, he worked for 27 years in air traffic control. He has a Master of Aeronautical Science in Operations from Embry-Riddle Aeronautical University.
Clyde Rinkinen is an Associate Professor at Embry-Riddle Aeronautical University, teaching Air Traffic Management (ATM) courses. He has worked in the ATM field for 31 years. Professor Rinkinen is also a pilot and a member of the FAA Safety Team (FAAST). He has a Master of Aeronautical Science in Educational Technology and Safety from Embry-Riddle Aeronautical University and a BS from Weber State University.
Luis F. Ramirez currently works to support training initiatives at a leading simulation and training contractor. He earned a BS in Psychology from the University of Central Florida with a focus on industrial/organizational psychology. While attending UCF, he also gained 2 years of research experience working at the Institute for Simulation and Training.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
