Abstract
Cognitive engineering methods must be extended to allow for analysis of educational systems. Specifically, this article proposes work action analysis (WAA) as a method for modeling educational systems to support their design and evaluation. WAA is defined as a form of work analysis that describes a system as the overall system goals, the collective work environments for each of multiple roles in the system, the essential interrelationships between the agents’ cognitive and physical activities and their environment, and the interactions between agents’ roles. A general description of WAA is given along with specific steps to generate a WAA model. WAA is used to model an undergraduate engineering course and guide its evaluation via a variety of measures. Finally, insights into educational systems from the WAA modeling method are discussed.
Introduction
Applicability of Cognitive Engineering to Educational Systems
Education is a domain relatively unexamined by cognitive engineering. Some of the earliest calls for the discipline of cognitive engineering observed its potential application to education (Norman, 1980). Since then, research has examined how methods such as work domain analysis (WDA) can be used to design curricula by identifying important aspects of a work domain that warrant instruction (Dainoff, Mark, Hall, & Richardson, 2002; Lintern & Naikar, 1998; Naikar & Sanderson, 1999, 2003). However, cognitive engineering methods have not been applied to study educational systems themselves. Thus, this article proposes work action analysis (WAA) as a method for modeling educational systems to support their design and evaluation.
To lay the foundation for this development, we define an educational system as any situation in which a person is going through a process of learning that is facilitated by another, such as one-on-one tutoring, a single classroom or course, or an entire institution. Brown (1992) observes that the educational systems of classrooms comprise complex and highly interrelated domains whereby learning takes place through a variety of activities that are interdependent and cumulative:
Classroom life is synergistic: Aspects of it that are often treated independently . . . actually form part of a systemic whole. Just as it is impossible to change one aspect of the system without creating perturbations in others, so too it is difficult to study any one aspect independently of the whole operating system. (Brown, 1992, pp. 179–180)
To evaluate educational systems, methods such as program theory evaluation have identified the need for a model of system behavior.
Evaluations which are based on program theory have two essential elements: an explicit model of the program (in particular, the mechanisms by which program activities are understood to contribute to the intended outcomes) and an evaluation [a set of measures, procedures, and analysis] which is guided by this model. (Rogers, 2000, p. 209)
However, the models used in program theory evaluation are fairly simple: They represent system elements (either context or mechanisms) as boxes, with arrows indicating causality between them. Likewise, as seen in examples in Rogers (2000), there is a lack of formal definition in creating these models, resulting in little differentiation between types of system elements and a narrow focus on certain portions of the system.
In contrast, WDA is an existing method that provides a formal method for creating a model of complex, dynamic systems. Likewise, WDA has a number of other qualities that make it suitable for examining education. Although many other cognitive engineering modeling methods emphasize a hierarchical or sequential pattern of activity, education is by nature more fluid; people have freedom to act and change their pattern of activity as appropriate to teaching and learning. Furthermore, WDA has been used to model work domains with multiple interacting roles, which will be considered later in greater detail.
Thus, cognitive engineering methods such as WDA can contribute the broader systems approach required to rigorously design and evaluate educational systems. However, this approach will require some adaptation from common practice. The most extensive application of WDA has been to systems in which the primary goal is described in terms of physical properties, (e.g., Bisantz & Vicente, 1994; Hajdukiewicz, Doyle, Milgram, Vicente, & Burns, 1998), commonly referred to as causal systems (see Bisantz & Ockerman, 2002, for discussion). In addition, WDA has been used, albeit less commonly, to examine systems structured more by social and organizational practices, referred to as intentional systems (Bisantz & Ockerman, 2002).
Although educational systems may be classified as intentional, they have several important characteristics. First, in contrast to other intentional systems, such as military command and control (Bisantz et al., 2003; Hajdukiewicz, Burns, Vicente, & Eggleston, 1999), a library (Rasmussen, Pejtersen, & Goodstein, 1994), intelligence analysis (Lintern, 2006), and analysis of nursing practices (Burns, Enomoto, & Momtahan, 2009), education systems are primarily concerned with internal cognitive change: The system goals center on neither physical outcomes nor outcomes shared across the organization but instead on students’ learning. This learning results from cognitive activity that is sensitive to the social and physical environment but also has its own internal processes and constraints. Thus, modeling educational systems requires identifying changes in students’ cognition as it evolves through the actions of the system agents and the dynamics of their work environments.
Second, educational systems include multiple human roles, including instructors and students, wherein the constructs of “role” and “agent” should be distinguished. The definition of agent used here is “an entity (either computer, or human) that is capable of carrying out goals, and is part of a larger community of agents that have mutual influence on each other” (Hayes, 1999, p. 127, emphasis in original). On the other hand, a role is an aggregate construct defined as a collection of agents with a common set of goals and means to carry them out. Thus, agents within a common role may themselves vary in their specific capabilities and personal objectives, even as they have a common view of their work environment. Historically, social, organizational, and cooperation analysis has assumed that a single WDA is sufficient for describing the work of all roles and their goals. For example, Hajdukiewicz (1998) performed a WDA to form a single abstraction hierarchy on which he then overlaid the regions of responsibility for which each role was responsible. In contrast, in educational systems, each role may have a significantly different view of the system and have disparate, sometimes-conflicting goals, even though their interactions are a fundamental driver of system performance. Additionally, in educational systems, some roles establish the work domain of other roles. For example, student behavior is driven by the environment created for them by the instructor, and the instructor’s behavior may change on the basis of feedback and analysis of students’ ongoing performance. Independent from our work, Burns, Bryant, and Chalmers (2005) recognized an analogous situation to educational systems in naval command and control, whereby opposing naval ships attempted to exert control over each other and applied separate abstraction hierarchies for agents that interact only at the physical level. Models of educational systems must further describe how one role may not only establish parameters for another but even fundamentally structure another role’s work environments and required behaviors to meet objectives, such as instructors’ decisions when defining assignments.
Third, the physical and social components of educational systems can change fluidly and rapidly; for example, an instructor may switch classrooms, textbooks, assignments, pedagogies, classroom procedures, and educational technology at any time. Again, Burns et al. (2005) gave an analogous situation in which agents (other ships) may enter and leave the system at various points, illustrating the changing nature of the system. However, unlike naval command and control, whereby each ship is essentially in a fixed configuration, an educational system can be reconfigured significantly. Thus, the behavior of each agent determines, to varying degrees, the constraints and affordances available in others’ work environments.
Fourth, the agents in the system may not be experts in their work environment. Students may take courses, and instructors give courses, in unfamiliar surroundings and using unfamiliar technology. Also, neither may be trained in the specific teaching and learning tasks they must perform. Furthermore, by the nature of an educational system, the students are there to learn and so are typically not skilled and knowledgeable in the skills and topics covered. Thus, the model must accommodate agents in the system who have poor strategies for working within the affordances and constraints of their environment to achieve their objectives.
To address these unique aspects of educational systems, this article proposes WAA as a method for modeling educational systems to support their design and evaluation, summarizing and building on an earlier, unpublished dissertation (Nickles, 2004). WAA is defined as a form of work analysis that describes a system as the overall system goals, the collective work environments for each of multiple roles in the system, the essential interrelationships between the agents’ cognitive and physical activities and their environment, and the interactions between the roles. The term action indicates that its focus is on the analysis of actions taken by agents as they interact with each other through their environment. A general description of WAA is given in the next section along with specific steps to generate a WAA model. Then, as a demonstration, WAA is used to model an undergraduate engineering course and guide its evaluation via a variety of measures. Finally, insights into educational systems from the WAA modeling method are discussed.
Extending Cognitive Engineering Methods to Educational Systems
Models of educational systems must describe the overall system goals, the work environment and objectives of each role in the system, the essential interrelationships between the agents’ cognitive and physical activities and their environment, and interactions between the roles. Like other cognitive engineering models, WAA models can be both descriptive and formative. When used as a formative model to guide design, a WAA model represents an anticipated set of feasible behaviors and explicitly describes how their design supports the system goals; this description captures the designer’s intentions in an explicit and systematic manner. When evaluating an established educational system, a WAA model can be used in a descriptive sense to refine an understanding of how the system actually functions and to make sense of disparate measures.
The WAA framework consists of three dimensions: (a) means-end, (b) parts-whole, and (c) roles of cognitive agents. As in WDA, the first two are hierarchical in nature, whereas the last is categorical. A schematic diagram of the WAA framework is shown in Figure 1, and the following subsections detail each of the dimensions.
Schematic diagram of the work action analysis framework.
Means-End Decomposition
WAA models are structured to address key attributes of educational systems. Some key distinctions are shared with WDA. First, each abstraction comprises a comprehensive view of the system. Second, relations between levels of abstraction are based on means-end relations. For any item at one level of abstraction, the related items at the level immediately below (less abstract) identify the means of accomplishing it, and the related items at the level immediately above (more abstract) identify the ends for which it is undertaken.
However, one significant change from WDA is in the nature of the links between levels used within the abstraction-decomposition. WDA specifically uses structural means-end relations, whereby links between elements describe fundamental constraints in the environment. “A WDA represents the thing being acted on . . . work domains are objects of action” (Vicente, 1999, p. 162). In describing the work domain, humans’ actions are assumed to be expert responses to the constraints inherent to the environment rather than constructs represented within the model.
In contrast, educational systems involve learning, which itself is an internal cognitive outcome resulting from cumulative physical and cognitive activities. Of note, cognitive learning activities are intrinsically linked to physical activities and outcomes through both mundane and generative processes: mundane when physical activities such as printing out a homework assignment must precede cognitive activities; more generative when physical activities such as reading, manipulating materials, and sketching diagrams embody the cognition of learning. Some specific pairings between physical actions and cognitive activities are provided in Table 1.
To distinguish from the structural means-end relationships used in WDA, the means-end relation used in WAA is defined as an agent-environment means-end relation. The term agent-environment signifies that these are means-end relations between artifacts and information in the work environment and the agent’s physical actions, cognitive activities, and goals, emphasizing their interrelationship.
Pairings of Some Student Cognitive Activities and Physical Actions
Four levels of abstraction are used in WAA models, as shown in Figure 1. These levels, going from bottom to top, abstract from the physical realm to each role’s objectives through the physical actions performed on the physical realm and the cognitive activities that bridge the actions and objectives. Specifically, the lowest level of abstraction is called artifacts and information. This level is analogous to the physical form level in WDA’s structural means-end abstraction hierarchy (Rasmussen, 1985), but in WAA, the level is broadened to include other types of resources, such as electronic files and items of information, that enable and constrain action but are not necessarily physical. This definition is within the original intent for this level in the abstraction hierarchy, as it identifies the resources required for the actions to be performed and the structures they create in the work environment (Rasmussen, 1985).
The level of abstraction above artifacts and information identifies physical actions, which are defined as the physical behavior performed on and with the artifacts and information. This behavior includes observable manipulations, such as typing on a keyboard, reading a book, giving a presentation to an audience, and playing an instrument. Physical actions are linked to artifacts and information by agent-environment means-end relations. The artifacts and information are indicated as the necessary means to accomplish the physical actions.
Jumping to the top of the hierarchy, a description of role objectives composes the highest level of abstraction. Ultimately, within each role, all system elements at other levels of abstraction should be means of achieving the objectives and thus are connected to them through the means-end relations. The role objectives level is intended to be descriptive, that is, it should capture what each role is actually striving to accomplish. In addition, it is often useful to include the normative system goals that the designer intends for the emergent system behavior to achieve. System goals may be matched with some of the roles’ objectives, but many roles may have separate objectives instead of, or in addition to, the system goals; thus we recommend the system goals be shown separately in a WAA model, as illustrated in Figure 1.
The most novel introduction to the WAA model is the inclusion of cognitive activities as an abstraction between physical actions and role objectives. The interplay between cognitive activity and physical action requires they be adjacent: The environment is structured to overtly require those physical actions that foster the cognitive activities required to meet role objectives. For example, a student writing a paper is concurrently performing the physical action of writing and the cognitive activity of constructing an argument. Although cognitive activities and physical actions are tightly connected, they can be distinguished as the internal cognitive activities of the mind and the externally observable actions of the body. Some activities may be difficult to precisely place in one category or the other. Reading, for example, could be classified in either level depending on whether the analyst considers the term to refer to the physical aspects of reading or the cognitive. Thus, the analyst must apply discretion to form a model that is both suitable for the analysis of a given system and feasible within the data available to the analyst, which may not, in many educational settings, allow for detailed observations of all student activities.
As the use of agent-environment means-end relations does not fall under the definition of WDA, their use in the abstraction hierarchy drove the naming of this method as work action analysis. The term work refers to its being a form of work analysis, as defined by Vicente (1999). The term action refers to the centrality of actions (both physical and cognitive) in this domain not only as the means to an end but also as the frequent definition of system goals that may seek, for example, that students apply specific cognitive activities to the topics being taught.
Parts-Whole Decomposition
As in WDA, the parts-whole decomposition is used here to break down larger system elements into smaller levels of subsystems. Granularity is a significant issue, as it is often necessary to examine the system both as a whole and at an appropriate level of detail for the purpose of the analysis. The number and content of the levels of the parts-whole dimension must be set on the basis of natural divisions within the domain and the needs of the designer or analyst.
In the educational systems studied by WAA, the parts-whole dimension is not a decomposition based solely on the physical environment but instead also identifies coherent educational units. For example, in university education, students generally follow a sequence of courses, with each course containing many units or modules. These educational units often have a temporal relation in that they must be performed in a sequence that develops prerequisite knowledge and skills. The specific decomposition levels will depend on the scope of the educational system and the subdivisions most relevant to the evaluation.
Cognitive Agent Roles Decomposition
To remain tractable, WAA does not seek to model individuals; instead, we recognize that “when agents have specialized functions they are said to have individual roles, such as pilot, navigator, or mechanic” (Hayes, 1999, p. 127, emphasis in original). Each role may perform similar work on some shared system elements but may also interact with different system elements, perform different tasks, and have their own objectives that may or may not align with the system goals. In addition, one role may create the environment of other roles, such as instructors establishing the environment for their students by creating the assignments and grading formula. The exact choice of roles will depend on the purpose of the analysis. In many educational systems, the dominant roles are the instructor and the student shown earlier in Figure 1; some analyses may choose to include other roles as relevant—perhaps teaching assistants or tutors in university education or parents overseeing the homework of primary school children.
In WAA, the domain of each separate role of agents is modeled in its own two dimensional means-end and parts-whole framework, similar to that of Burns et al. (2005). In addition, the overall system goals are explicitly represented outside the models of the agents’ environment. It must be recognized that the system goals (such as the learning objectives of a course) may not be shared by all students, some of whom may instead be focused on grades or other objectives.
Although the roles are distinct, they are not isolated from each other. A major extension here from WDA is the addition of correspondence relations, capturing the specific points of contact between the frameworks for each role to show how they influence each other and work together to accomplish the system goals. Correspondence relations can also identify where a role’s objectives align with the system goals. In Figure 1, correspondence relations are indicated by the arrows connecting elements of each individual role’s two-dimensional framework or goals to those for another role.
Our focus in this article is on university education, in which many of the activities of the instructor and student are comparatively distal and asynchronous. For example, although their physical actions and cognitive activities hopefully have some fruitful interplay during “contact hours,” such as lectures and office hours, the majority of the students’ time is spent on self-directed activities away from the instructor, including reading and completing assignments, many of which cannot be directly observed. In such situations, correspondence relations between agents are generally at the lowest level: The instructor provides information and artifacts to the students (such as lecture material and assignments), and these drive the students’ actions, as illustrated in Figure 1. For example, if a textbook is used in a course, it is shown as an artifact for both the instructor and student, linked between the two roles via a correspondence relation. In other contexts, correspondence relations between agents may occur at higher levels of abstraction.
These correspondence relations can be used to trace how system goals are fostered not only within a role but between roles. One role can influence another by creating or specifying artifacts and information for other roles, on which the physical actions and cognitive activities of the other roles are enacted. For example, an instructor’s role objectives may have a correspondence relation with the system goals for learning; the students’ role objectives may focus on receiving a high grade. Thus, the instructor must design the course to motivate students via grades to perform the necessary cognitive activities for learning. To this end, the instructor creates artifacts and information, such as syllabi with grading standards and assignments, and makes them part of the students’ environment as a means of structuring the students’ behavior.
Procedure for Creating a WAA Model
A general procedure for creating a WAA model is presented here. This procedure is specific to WAA but is based on WDA methods as given by Rasmussen et al. (1994) and Vicente (1999). As in WDA, these methods are intended to serve as guidelines, as there may be specific needs for particular domains and tasks. Also, these guidelines should not be followed in a strictly sequential manner; each step is intended to provide insight about the system, which in turn leads to refinements to the framework and model established in previous steps.
Method
1. Determine the scope and purpose of the analysis
Both the scope of the system to be examined and the purpose of the analysis must be specified to define the context for the analysis. These will serve as boundaries and guides to development of the framework and creation of the model and will determine the appropriate level of detail.
2. Determine the system goals
For many types of analysis, the goals of the system should be identified to ascertain the normative standards for system performance. It must be recognized here that the learning goals must be sufficiently specified for design and evaluation, which is often not the case in, for example, university education (St. Clair & Baker, 2000). Specifically, the learning goals should point to cognitive activities representative of the desired depth of learning, including the extent to which students should be able to abstract and apply their new knowledge.
3. Identify all the roles of cognitive agents that are integral to the system
This step should be relatively simple when the roles are clearly delineated (e.g., student and instructor) relative to the purpose and scope of the analysis.
4. Identify the levels of the parts-whole and means-end dimensions
There may be an established system of division into components that can be used to design the parts-whole dimension, such as course or module structures; similarly, temporal relations, such as assignment deadlines and exam dates, identify temporally related educational units. In parallel with the parts-whole dimension, the definition of each level of the means-end dimension should be determined. Although the four levels of the means-end dimension identified here are artifacts and information, physical actions, cognitive activities, and role objectives, some domains may require deviation from these general categories and/or different numbers of levels to analyze a particular system for a particular purpose. The schematic framework presented in Figure 1 with defined categories in each dimension is an example of a final product of this stage.
5. For each role, fill in the items at the lowest and highest levels on the means-end and parts-whole dimensions so that the top left and bottom right corners of the framework are populated
Having defined the categories in each dimension of the framework, this stage then populates those elements that are the most tangible and categorical and that can typically be identified unambiguously at this point. Identifying these items also keeps the model appropriately bounded.
6. For each role, fill in the items in all other levels, identifying relations between levels as appropriate
After the elements from the previous step are specified, elements of levels in between can be identified by their level of abstraction, location within the part-whole dimension, and correspondence to elements in models of other roles. At this point, the how and why questions must be used to determine whether items at different levels of abstraction are properly related by means-end relations; if two items in adjacent levels of abstraction are related, the one at the lower level will identify how the other is accomplished, and the one at the higher level will identify why the other is performed. Parts-whole relations must also be identified; they specify the items that are a part of a larger whole and identify temporal relations between elements that may require representation as separate parts. This process will also suggest new system elements by making the analyst consider all the system elements that may be means to an end and ends of a means, or parts of the whole. Of particular note in educational systems, this stage calls on insights from educational practice and the learning sciences to identify which cognitive activities reflect the learning goals of the system, which physical actions will foster them, and which information and artifacts (in the context of students’ role objectives) will support them. For example, the instructor’s task of creating an assignment will foster learning goals, which are provided to the students as an artifact that requires the physical actions corresponding to desired cognitive activities and that motivates the students to perform this work because of some tie to their role objectives (hopefully, a desire to learn—but often a relationship to grades). In some cases, different sets of actions may be explicitly recognized as feasible, mirroring the recognition of multiple sets of allowable strategies during strategy analysis (Vicente, 1999).
7. Identify correspondence relations between role objectives and to system goals
Correspondence relations can exist between artifacts and information used by different roles and between role objectives and system goals. A well aligned system is characterized by the following:
All system goals have a correspondence relation with at least one role objective.
Roles with role objectives corresponding to system goals should have correspondence relations via artifacts and information with other roles contributing toward that goal.
All roles are related to the overall system goals either directly through correspondence relations between their role objectives and system goals or more indirectly via correspondence relations of their artifacts and information to roles whose role objectives correspond to system goals.
Framework Templates
Given the expense and expertise required to construct a WAA model (similar to constructing an abstraction hierarchy within WDA), after a model of one educational system has been developed, it can be used as a template for similar systems. For example, many undergraduate engineering courses follow a similar pattern: Students attend lecture, work through periodic assignments, take tests on the material, and receive graded feedback. The physical actions and cognitive activities performed in these courses are very similar to each other, so a set of templates can be made for homework assignments associated with particular learning goals and the type of material, lectures, and grading that is associated with each.
Although different courses are taught by different instructors with different content, they can benefit from a similar pool of templates centered on shared work practices and pedagogies. In university education, this role can be useful for education specialists: Different types of learning goals can be identified, such as distinctions between whether material should merely be memorized versus whether more extensive abstraction, comprehension, and application of the material is sought; and then the sequences of physical actions and cognitive activities required for each can be described, and supporting information and artifacts can be outlined. From this practice, university instructors, who are often untrained in educational methods, can be informed and guided to useful practices and the types of behaviors they should foster in students in general, recognizing the range of behaviors that may occur within the student role.
Also, when a new work practice is desired, templates can be created to represent and communicate it to instructors. In educational systems, this process would often be framed as new pedagogies corresponding to introduction of new educational technology or a new method of classroom instruction; such a template would allow a specialist in the pedagogy to represent it to instructors and curriculum designers.
Demonstration: WAA in Undergraduate Engineering Education
To demonstrate WAA as a method that can model educational systems and be used for their evaluation, we applied it to a senior-level industrial and systems engineering course on the design of human-machine systems. Here, we describe a WAA model of the portion of the course associated with the second homework assignment, which included both a critique of a design by each student and then peer review among students of their submissions. As the key assignments were similar, only one assignment in this series is described here as an exemplar. In addition, a set of measures was made on this system; the WAA model served as a framework by which to interpret a variety of measures made through the system.
Of the 53 students in the class, 49 (92%) gave informed consent for their data to be used for research purposes. The course instructor (the second author) presented lectures using PowerPoint presentations that were made available to students as topic files through a web-based course management system. A teaching assistant graded the second homework assignment and peer review comments, without awareness of the modeling and evaluation process.
Making the WAA Models
An evaluator (the first author, otherwise unconnected with the course) constructed the WAA model as described below in consultation with the course instructor. For clarity, this section is presented according to the linear sequence described earlier, although some iteration did occur. The resulting models are shown in Figures 2 and 3 for the student and instructor roles, respectively.
1. Determine the scope and purpose of the analysis
The scope of this analysis is the second homework assignment of the course described previously. The purpose of this analysis is to guide the evaluation of the instructional effectiveness of this portion of the course, with the aim of improving subsequent instruction.
2. Determine the system goals
The goals of the system are identified as the course goals as stated in the course syllabus. Of note to this unit was the course goal “Understand how we, as engineers, can design information systems to create effective work processes.”
3. Identify all the roles of cognitive agents that are integral to the system
The two roles of relevance here are instructor and student. Although there was a teaching assistant assigned to this course, her functions were to assist on a subset of the instructor’s duties, and so she is included in the instructor role.
4. Identify the levels of the parts-whole and means-end dimensions
Our decomposition in Figure 1 accords with the typical structure of undergraduate engineering courses; other decompositions may be better suited to other course formats. In this case, the most detailed aspect of the parts-whole decomposition is the individual topic of course content, that is, a single cohesive concept that students must learn as part of a course (Pritchett et al., 2002). Topics are typically associated with specific instructional material, which may include a section or chapter of a textbook, a lecture, and/or paper or electronic notes. In this course, there are typically three to four topics per class lecture. Learning each topic was assumed, during initial development of the course and the corresponding WAA model, to require a set of physical actions and cognitive activities. These actions may include reading and memorizing the topic material or applying the topic to a specific application to gain design experience. In some cases, different sets of actions were recognized as being possible, depending on individual student characteristics.
The next-largest level of the parts-whole dimension consists of assignments. Many undergraduate engineering courses are structured so that an assignment, such as a homework or quiz, covers one or more topics. Thus, a group of (often related) topics is covered by a single assignment. Grouping content on the basis of assignments corresponds to normal teaching activities, arising as much from pragmatic reasons as conceptual analysis of the course content: The instructor schedules topics partially on the basis of when they will fit in assignment due dates and partially to highlight a cohesive group of topics. Students also schedule and organize their work (i.e., physical actions and cognitive activities) in relation to the assignments. In this case, 15 assignments (12 homework, one midterm exam, one project, and one final exam) were listed in temporal order in the full model. Homework Assignments 2 through 12 asked students to identify a good design and a bad design, post a description on the course website of why they are good or bad in light of that week’s topics, and then provide peer-review comments of five other students’ postings. Homework 2 is related to a week’s discussion of four specific topics drawn from Beyer and Holtzblatt’s (1998) contextual inquiry. Only Homework 2 is discussed here as an example, but subsequent homework assignments built on it, and thus their analysis also built on the analysis illustrated here.
The next level in education is the course. In the context of this undergraduate engineering example, a course is a set of assignments corresponding to a set of topics with a consistent instructor (or instructors). This level is included to contextualize the individual homework assignment examined here.
5. For each role, fill in the items at the lowest and highest levels on the means-end and parts-whole dimensions so that the top left and bottom right corners of the framework are populated
This stage identified both the objectives of the roles (captured at the highest level of abstraction in the model) and the artifacts and information (described at the lowest level of abstraction in the most detailed parts-whole descriptions). For the instructor’s role, we assumed that the coursewide role objectives corresponded with the system goals. The student role may have several coursewide objectives: good grades, learning, career preparation, and effective use of time and resources. Individual students within the student role may place their own emphasis on each of these objectives, but the instructor needs to design for a student role that spans most of the individuals.
The artifacts and information level on the means-end decomposition and the content level of the parts-whole dimensions include any material, physical or electronic, that contains information on the content to be learned and related material to be communicated between the roles. The information communicated in the lecture is considered an element of information, as are in-class discussions, electronic slides used during lecture, and out-of-class student-instructor dialog. The instructor also provides an assignment to the students. In addition to these elements shared by both roles, students may take notes before, during, and after the lectures as study aids, and instructors may have personal lecture notes.
6. For each role, fill in the items in all other levels, identifying relations between levels as appropriate
This step consisted of two major activities performed iteratively until the evaluator was satisfied that the end result accurately represented the actual attributes of the course as known at that time. First, the evaluator populated the table with elements in the system. Second, the evaluator identified agent-environment means-end and parts-whole relations that exist between the elements, both as a product of the modeling process and as a check for correctness and completeness within the modeling process.
The students’ cognitive activities were initially identified by the evaluator working with the course instructor as an explicit representation of the instructor’s expectations for students’ learning processes and intellectual capabilities. Their physical actions were extrapolated from the cognitive activities and from a pragmatic assessment of the affordances and constraints provided by the artifacts and information, with particular regard to activities that were observable from logs of student interaction with the course website. The results of this step (without the agent-environment means-end relations, for legibility) are shown in Figure 2 for the student role and in Figure 3 for the instructor role.
7. Identify correspondence relations between roles’ artifacts and information and between role objectives and system goals
The roles of instructor and student share multiple artifacts and information at each parts-whole level in Homework 2, shown underlined in Figures 2 and 3. Each of these shared elements is identified as having a correspondence relation between the roles. As the instructor in this case is attempting to achieve the system goals, there are also correspondence relations between the instructor’s role objectives and the system goals. Similar correspondence relations will exist whenever student objectives intrinsically include learning and thus mirror the system goals. In other cases, the relationship between a student’s objectives to, say, achieve a high grade is linked to system goals only if the instructor effectively creates artifacts and information that structure the students’ environment such that activities meeting their objectives (grades) also foster learning.
Work action analysis model for Homework 2: Student role.
Work action analysis model for Homework 2: Instructor role.
Evaluation With the Use of WAA
Once a WAA model of the system was created, we used it to evaluate this educational system. First, the WAA model was used as a basis for selecting the measures to collect. Some elements cannot be feasibly measured (e.g., many learning activities happen outside of lecture hours), and many cognitive activities can only be inferred via measures of observable physical actions. This difficulty in measurement highlights the value of a model by which relationships may be traced through the system. The measures collected and used in our example are summarized in Table 2, grouped according to the type of measure.
Second, after the measures were collected, the evaluator made a judgment on what the measure indicated about the associated element of the model. This judgment was placed on the associated element in the WAA model. The evaluator then considered the measures in the context of the whole system by tracing through the relations within roles and correspondence relations between roles. The evaluator judged, whenever there was an unexpected discrepancy, whether it represented a problem within the system and the potential causes and implications of that problem. In some cases, the unexpected behavior was evaluated to not be a problem but instead highlighted another strategy that students were executing that could contribute to system goals, and the model was extended accordingly. Although an evaluator may already have a sense that there is a problem if any measure’s value is unexpected, the model provides the context and guidance to determine where that problem exists in the system and how its effects propagate throughout.
Measures Collected on Homework 2
The measures for our example are listed in Table 3. If only the measures are considered, the lower-than-expected grades suggest that there is a problem with both parts of the assignment (design critique and peer review) but could not easily disambiguate between several possible hypotheses: (a) the students did not develop the requisite knowledge on the topics, (b) the students did not have the skills to understand or fulfill the homework’s requirement to relate their knowledge to everyday systems and peer review others’ comments, (c) the students did not care to spend sufficient time on the assignment, or (d) students experienced technical difficulties in completing the homework (e.g., difficulty posting their descriptions and peer-review comments on the course management system).
Evaluation Measures for Homework 2
The WAA model of the student’s role, overlaid with the associated measures relative to the evaluator’s expectations, is shown in Figure 4. Nongraded test questions (“assessments”) provided via the course management system gave students an opportunity to test their knowledge after viewing a topic and also anonymously captured their knowledge state for the evaluator. Likewise, students were able to rate the instructional efficacy of the topic files. Scores on these assessments and ratings were high, suggesting that the students were able to develop the requisite knowledge and thus contradicting Hypothesis 1 given earlier. Furthermore, the responses to Survey Question 2 indicate that a majority of students believed it was “somewhat easy” to learn the material, and no students rated the material as “somewhat” or “very” difficult. The number of “hits” was high on several items: feedback from the instructor, topic files (especially after the due date), submissions of other students available for peer review, and peer reviews of their own submission. These measures show both that student interest in the assignment was high (contradicting Hypothesis 3) and that students were able to work within the course structure, including the course management system (contradicting Hypothesis 4). The problems, then, appear to be isolated to some of the cognitive activities associated with the assignment. Successful performance on this assignment required the cognitive activities “Evaluate designs relative to content” and “Evaluate others’ work.” The prerequisite activities of “Internalize knowledge” were found to be successful, and students were able to select a design to critique (as evidenced by having a posting at all), but their grades show they were not able to effectively evaluate their own designs and evaluate other students’ descriptions. Correspondingly, most students reported activities focusing more on understanding the topic than on relating it to new situations (Survey Question 3), and some did not complete the required number of peer reviews in the time given. This result supports Hypothesis 2, that students were not successful in these cognitive activities; further evidence comes from the high number of hits from students seeking more information from instructor feedback and from reviewing the topic files more after the assignment’s due date. From these insights, several solutions became apparent, directing the instructor to modify activities to provide more opportunities for students to develop their skills in these cognitive activities. The same type of homework assignment was given for the next several weeks based on different content, and prompt grading feedback was provided to students highlighting aspects of good and bad performance. Student grades improved across the next several assignments; by Homework Assignment 4, the average homework grade was a letter grade higher than that for Homework 2.
Work action analysis model of Homework 2 overlaid with measures relative to expectations.
Contributions of Work Action Analysis
Education is a complex sociotechnical system for which cognitive engineering methods should be developed to aid in its design and evaluation. Educational systems have certain qualities, including a focus on internal cognitive change, multiple interacting roles, work environments for each agent that are created and frequently modified by other agents, and possible lack of expertise, all of which make such systems difficult to examine with current cognitive engineering methods. Other sociotechnical systems may share these traits, and thus may also benefit from the WAA process and model framework developed here, such as hospital emergency rooms, sporting events, and courtroom proceedings. Each of these situations can be characterized as having many agents fulfilling multiple roles that interact and shape the space in which other agents act. In the emergency room, the attending physician directs the overall procedures so that agents in other roles act within the frameworks set by that direction. In American football, the configuration and play call of the offensive team sets frameworks in which each agent in each position (or role) on both offense and defense act. Courtroom proceedings involve multiple roles that are influenced by the actions of the attorneys, witnesses, and the presiding judge.
As noted previously, educational systems can be relatively small one-on-one situations or as large as entire schools. For larger educational systems, WAA models may be presented with less detail to keep the models tractable during design or evaluation, and for smaller educational systems, the models can be quite detailed to the point of fostering individualized instruction around models of each student.
Benefits to the Design and Evaluation of Education
Common approaches to the design and evaluation of education, such as end-of-semester course evaluations by students or occasional observations by expert educators, lack the detail and model basis provided by WAA. Thus, designers and evaluators of educational systems can gain several benefits from using the WAA process. First, as seen in the study of a portion of an undergraduate course, a WAA model with an appropriate set of measures can support the evaluator’s work in identifying specific system elements that need improvement. Although an end-of-semester student course evaluation may (belatedly) identify the general nature of a problem, and an experienced evaluator may be able to identify the root problems of a system without a WAA model, the model supports even a domain expert unfamiliar with design and evaluation methods to make a comprehensive and systematic representation of the system by which to make sense of disparate measures to identify problems and potential resolutions. For example, WAA has been used to model a professional development program for science teachers to evaluate the design of its web-based portal (Nickles, 2007).
Second, WAA provides an explicit model of the system that can be used to communicate a detailed, comprehensive design of the system to other designers and to those agents actually in the system. In the case of education, curriculum designers may construct or advise on both curriculum content and pedagogical method; WAA provides a model form for documenting their intended design at multiple levels of abstraction for themselves, for the instructors, and potentially, for the students to reference.
Third, changes in one aspect of an educational system are often analyzed in comparative isolation from other system components. Measures are then developed to evaluate the expected impact of this one isolated change, and some, such as student evaluations, may themselves have biases or limited perspectives. In contrast, through the process of making a comprehensive WAA model, the designer has a systematic process for considering multiple aspects of the system design, for anticipating how changes may propagate through the system, and for catching potential problems that could otherwise be overlooked.
Fourth, the process of creating a WAA model can be used to explicitly test the alignment of the system. Alignment refers to how well the elements of an educational system are selected and connected to produce the desired goals. Although it is considered desirable in education (Bransford, Brown, & Cocking, 2000), there is little guidance in the literature for evaluating and designing for effective alignment within an educational system. Using the WAA process, an evaluator can determine whether a system element supports the system goals directly or whether it is related to the goals indirectly via means-end (within agent) and correspondence relations (between agents and between agent objectives and system goals) or does not appear to support system performance. The system is well aligned when all elements are related by means-end relations to role objectives and when roles that explicitly attempt to achieve the system goals influence other roles to that end via correspondence relations. In our example, the system goals centered on student learning are striven for not just by students’ joy of learning but also through the instructor role of creating a course structure and grading formula that uses students’ desire for good grades to motivate them to undertake cognitive activities that foster learning.
Fifth, the evaluator has a complete chain of means-end and parts-whole relations that shows how any single element is related to the course objectives. This chain allows the evaluator to speculate how a failure at one element of the system could lead to a break in the sequence of elements that support a course objective. Also, the evaluator can determine how much redundancy is in the system by examining how many independent means-end chains lead to the same goals, where a greater number of independent chains increases the likelihood that an individual goal will be met and will allow for idiosyncratic agent behaviors within roles. An added benefit to the means-end relations is the ability to examine the path between an individual element and system goals in an explicit, structured manner. This ability allows the evaluator to determine not only whether an individual element is a means to achieve the system goals but also how direct the linkages are.
Insights on Educational Systems
The development of the WAA modeling framework also provides several insights on educational systems that inform further work in the cognitive engineering paradigm. First, WAA provides conceptual insights about the merger of cognitivist and ecological approaches to system modeling. Although agents in a work environment are driven by both cognitive and ecological constraints, work analysis typically models them separately, sometimes even sparking debates as to which should be examined first. WAA simultaneously models the influence of both on the behavior found in educational systems. To represent the influence of cognitive constraints, cognitive activities (which are related to role objectives) are described as being concurrent with corresponding physical actions. To capture ecological influences, the artifacts and information in the system proximally constrain the physical actions performed and, through physical actions, influence cognitive activities. Although the WAA representation does not attempt to quantify the comparative influence of cognitive and ecological motivations on an agent, it does allow the modeler to trace how particular agent behavior is influenced by internal motivation and external constraint via the agent-environment means-end relations.
Second, interaction and influence between agents is a critical component of many systems. In the WAA framework, agent roles can be distinguished while showing specific pathways by which these roles interact and influence each other. This aspect contrasts with other cognitive engineering methods that do not distinguish roles, separate them without showing their interactions, or assume that they are operating as peers with shared goals and operating on a shared environment. The WAA model also shows that artifacts and information are often the only means that roles have to directly influence each other. This aspect emphasizes the importance of the design of these artifacts and information from an organizational perspective.
A third insight of WAA is that in some domains, agents can create and shape each other’s work environment to the point of defining and shaping the constraints in which an agent works. Thus, in WAA, the work environment can be viewed as a more fluid concept than in methods wherein the work environment is based on fixed physical structures. In a multiagent system, agents may have different types of influences on each other at different times, in effect changing each other’s work environment. In such systems, this dynamic helps describe how the system can evolve through time, adapting toward higher performance standards and in concert with changes in agent capabilities (e.g., student learning). From a design and evaluation perspective, this dynamic must be captured within the models. From an operational perspective, agents attempting to influence other agents benefit from better understanding not just of their own work environment but of other agents’ environments and objectives.
A final insight on multiagent systems is the separation of system goals from agent objectives. In our analysis, we assumed that the system was aligned, that is, at least one role was attempting to meet the system’s designated goals and directing other agents toward that end. Within that assumption, WAA provides a process for tracing out the effectiveness and extent of a system’s alignment. However, individual agents have their own objectives that may or may not be in pursuit of the stated system goals. Making this separation allows a richer description of the system, showing which agents are attempting to meet the system goals and how they influence other agents to bring them about. At an extreme, if no roles are explicitly pursuing system goals, then system performance may be viewed as an emergent property of the agents’ combined cognitive activities and physical actions; although the WAA model represents these dynamics, further analysis would be required to see whether they would collectively generate the desired system-level performance while pursuing their own objectives.