Evaluation of Strategic Emergency Response Training on an OLIVE Platform

Abstract

Virtual worlds provide a new methodological framework for conducting emergency response exercises. The research attempts to evaluate whether the use of virtual worlds avoids recognized weaknesses of traditional emergency response exercises and facilitates further learning outcomes. The objectives of the article include, first, to examine whether the virtual worlds contribute to learning experiences for the emergency response exercises; second, to use a case study to evaluate and synthesize preliminary findings of the strategic flood response exercise; and third, to make recommendations for conducting a larger scale emergency response in an online Virtual Learning Environment. The preliminary findings suggest that the use of virtual worlds could potentially facilitate better learning outcomes and provide as an effective training methodology to the emergency communities. It also identifies the challenges of developing virtual world exercises, for instance, the need for better collaboration and communication between developers, academics, and end users.

Keywords

communication competence building coordination debriefing decision-making emergency response evaluation exercise scenario flood management learning outcomes OLIVE simulation effectiveness strategic exercise team training virtual worlds

The evolution of technology and web-service has provided developers and users with new applications to e-learning, for example, the use of virtual worlds. Virtual worlds refer to the persistent, computer-mediated environments where players can interact with the “world” and other players (Bartle, 2006, p. 31). With the multimodal nature of virtual worlds and the functions they serve, such as sharing space, resource and ideas, and highly interactive online technologies and reflective exercises, virtual worlds have become a new methodology for pedagogical purpose and facilitate learning outcomes (de Freitas, 2008; Liarokapis, Macan, Malone, Rebolledo-Mendez, & de Freitas, 2009). Practitioners and researchers have also started to use virtual worlds as an alternative methodology to conduct emergency response exercises in order to avoid the limitations of traditional emergency response exercises, such as not being cost effective and not being able to reuse the exercises (Chen, Parker, & Townsend, 2009).

Although several virtual world games and exercises have been developed to train emergency service personnel, only a few exercises that aim to train strategic managers to respond to large-scale emergencies have been developed. The research team have designed a national scale flood scenario to enhance the capabilities and skills of strategic managers. The project uses a Forterra Systems’ On-Line Interactive Virtual Environment (OLIVE^™) platform to provide an immersive 3D virtual learning environment with the principle aiming to develop the strategic decision-making skills required to manage a major scale flood emergency.

The objectives of the article include, first, to examine whether the virtual worlds contribute to learning experiences for the emergency response exercises; second, to use a case study to evaluate and synthesize preliminary findings of the strategic flood response exercise; and third, to make recommendations for conducting a larger scale emergency response in an online Virtual Learning Environment.

The preliminary findings suggest that the use of virtual worlds could potentially facilitate better learning outcomes and provide as an effective training methodology to the emergency communities. It also identifies challenges of developing virtual world exercises, for instance, the need for better collaboration and communication between developers, academics, and end users.

E-Learning and Virtual Worlds

E-learning can be defined as “the use of electronic multimedia technology to deliver education, information skills, knowledge and individual learning programs to large audiences, using the internet, intranet and other technology based systems” (Davies, 200, p. 19). It has become one of the most powerful techniques to facilitate students’ learning outcomes. The advanced digital technologies and computerized delivery methods increase the interests to use e-learning (electronic learning) to deliver various kinds of teaching and learning. E-learning is believed to be more effective than traditional learning (Asan, 2003; Cole & Hilliard, 2006; Connolly & Stansfield, 2006; deLeon & Killian, 2000) despite some controversies (Shen, Wang, & Shen, 2009).

The advantages of e-learning comparing traditional learning include (a) more methods of teaching and learning, (b) meeting students’ diverse needs, (c) learning from playing (at home), (d) learner-centered environment (Collis, 1996; Higher Education Funding Council for England (HEFCE), 2005; Westera, 1999), and (e) flexible, cost-saving, and more efficient use of time and staff motivation. In addition to these advantages, Alexander (2001) further argues that using technology in both classroom and distance learning has the following advantages: (a) increasing the possibility and opportunity to get access to education and training, (b) reducing the cost of education, and (c) improving the cost-effectiveness of education.

What is more significant is that e-learning also poses a potential advantage: it facilitates lifelong learning. People can have access through the Internet to the learning materials any time and everywhere. This has significant meaning for workplace learning. It is believed that e-learning increases productivity and profitability, and enhances employee loyalty (Gunasekaran, McNeil, & Shaul, 2002).

As a second generation of e-learning, the use of virtual worlds further facilitates learners’ learning outcomes by using highly interactive online technologies and reflective exercises (Chen, Rebolledo-Mendez, Liarokapis, de Freitas, & Parker, 2008; Connolly & Stansfield, 2006; de Freitas, 2008). Learning will occur more effectively within these environments if participants feel a sense of presence (Biocca, 1997).

Participants are represented in the virtual world through avatars (in-world representations of the trainee). The application also supports a range of functionality that includes supporting social interactions, modeling real environments, document sharing, and recording facilities that allow users to replay to activities undertaken in-world. Moreover, the 3D art assets, including avatar clothing, gestures, and faces, as well as buildings and vehicles, enhance the feeling of total immersion.

The multimodal dimension of virtual worlds has indicated positive benefits for learning transfer, and a number of leading edge projects using virtual worlds for supporting learning have suggested that virtual worlds can be very effective for training purposes, for example, Youngblood et al. (2007).

An immersive virtual environment provides participants with an insight into those roles that could not be attained through simply assimilating information about their role. However, the ability to fully inhabit those roles could be limited by physical reality, both that of the participants not resembling the roles they are taking on and the environment within which they are role-playing being too normal and familiar.

Some advantages of virtual worlds over traditional methods include the possibility of conducting real-time, distributed, multiuser, immersive, collaborative, and interaction modes. The attributes such as shared space, graphic user interface, immediacy, interactivity, and persistence may attract learners’ attention when it is related to learning activities. However, it is not certain whether these attributes are attractive to the learning and training for emergency responses. The development of an evaluation methodology for these virtual world platforms is required.

Halkett (2002) identifies that effective e-learning must be accompanied with the following two factors. First is the centrality of the tutor. With the development of new tools and methods used in the e-learning, trainers are expected to learn these skills; it is not possible for a trainer to use e-lectures, chat rooms, or interactive assessments “automatically.” Second is the need to master the details. Trainers not only need to understand how to use this new technology, but they also need to master these new methods so that they will be able to conduct their teaching more effectively. They should plan their lessons/curriculum and assessments based on these new elements.

E-Learning in Emergency Service Training and Exercises

From a pedagogical perspective, training exercises address the adult learners’ preference for active and experience-centered learning and opportunities for a more inductive or guided discovery learning approach. Emergency services exercises help to develop participants’ competencies and facilitate teamwork skills, and intra- and interagency communication and coordination (Cabinet Office, 2013; Ford & Schmidt, 2000, p. 197; Schaafstal, Johnston, & Oser, 2001; Stolk, Alexandrian, Gros, & Paggio, 2001). They provide a platform for participants to develop a more detailed understanding of individual participants and agency groups’ response culture, including decision criteria, processes, cultures, and formal and informal communication structures (Boin, Kofman-Bos, & Overdijk, 2004).

Scholars have addressed several challenges faced by these emergency response exercises. First of all, strategic exercises normally involve more abstract and complex tasks. This has prevented the setting up of clear learning outcomes and evaluation (Chen et al., 2008; Lee, Trim, Upton, & Upton, 2010). Next, the challenges of these exercises include the limited training time available and, probably, the inadequate funds. It also requires large numbers of personnel, for example, exercise planners, participants, observers, and training technologists (Ford & Schmidt, 2000; Schaafstal et al., 2001). As a result, it is difficult to conduct frequent exercises. Pedagogical challenges of emergency service training exercises also include retention, generalization, and progressive training (Ford & Schmidt, 2000). Other challenges include increasing flexibility, fidelity, validity (Green, 2000), and evaluation (Burrow & Berardinelli, 2003).

To overcome the difficulties of conducting traditional training exercises, computer-based simulations have been widely used by emergency response agencies to facilitate learning activities and learning outcomes (e.g., Granlund, 2001; Michael & Chen, 2005; Robert, Gamelin, Hausler, & Jarry, 1996). A number of “hybrid” tabletop-computer-based exercise scenario systems such as Emergency Command System and Tactical Command Trainer–Fire developed by Vector Command Systems are good examples (Vector Command, 2010). The former uses a traditional tabletop scenario approach but facilitates the integration of video feeds, 3D simulations, GIS mapping, information management, and communication technologies. The latter system makes use of a gaming approach where participants role-play tactical level decision makers, and an artificial intelligence system is used to direct fire at personnel behavior on the incident ground.

Due to the rigid system design, it is more difficult, more complex, and more expensive for computer systems to simulate overall scenarios to the required degree of fidelity, and consequently, factors involved in the scenario may be over simplified (Stolk et al., 2001). Current computer-based simulations have not maximized the potential usage of the game. For example, participants are still required to travel to an appointed place to take part in the computer-based exercise. It is time-consuming, and it is opposite to the ideal of e-learning. Bramley (1991) and Lim, Lee, and Nam (2007) argue that effective vocational training should take place in an authentic working environment.

Exercise ATLANTIS

Emergency exercise scenario as a training tool has been widely used by the emergency response training community (McCreight 2011). It is well established that scenario methodology goes a significant way to bridge the gap between concepts and processes in a classroom-based training and the practical experience of real emergencies (Alexander, 2000). Exercises herein refer to a training method that requires high participation and explicit learning objectives.

The general scenario for the exercise ATLANTIS was based on a one-in-a-100-year flood and the flooding events of 2007 in the United Kingdom. The impact of the flood event met the H20 Flood Risk category within the U.K. National Risk Register. The story line linked the research explicitly with the national flood exercise held by the U.K. Cabinet Office and Department of Environment, Food, and Rural Affairs’ national flood exercise in March 2011. Five essential skills required to respond to major flood disasters were identified, including communication, understanding regulations and response strategies, coordination, strategic decision-making, and teamwork skills.

The story line for the exercise was divided into two phases: pre-flood during the flood warning phase and then at the peak of the flood event. In Phase 1, participants were required to make strategic decisions during the “flood warning” phase, primarily focusing on the proactive evacuation of vulnerable populations and the protection of critical infrastructure. In Phase 2, during peak flooding, the scenario became more complex as participants were asked to shift from preparedness to response. Injects were developed to facilitate their decision-making and aligned skills associated with the rescue and care of displaced public and also dealing with critical incidents related to the flooding, such as water contamination, HazMat pollution event, and the potential loss of electricity across the region. “Inject” refers to something new to the exercise scenario and is required to be dealt with during the exercise. This term is used in the U.K. emergency service exercises. Participants played representatives from several organizations that were involved in responding to a major flood hazard, for example, local authority, Environment Agency, the emergency services, and utility companies.

The Use of the OLIVE Platform

On-Line Interactive Virtual Environment (OLIVE) is a software platform that allows customers, partners, and developers to create persistent virtual worlds where users can collaborate over networks to communicate, train, rehearse, analyze, experiment, socialize, and entertain (Forterra Systems, 2007). OLIVE employs a client-server architecture where PC clients are connected to a central server via a network. The architecture ranges from single user applications in one physical location to large-scale, simulated environments supporting many thousands of concurrent as well as geographically distributed users. OLIVE supports a great number of capabilities and functionality in the baseline platform, which are essential to support a wide variety of interactive virtual world operations. This includes fully operational avatars, voice over IP communication, distributed physics, networking, and a session record and playback capability. Moreover, a set of general 3D art assets including avatar clothing, gestures, faces, as well as buildings, vehicles, vegetation, and many other objects are provided. It is worth mentioning that external parties can create and control their own virtual worlds through the OLIVE SDK, through licensing (Chen et al., 2008). To summarize, it seems that OLIVE is suitable to be considered as an appropriate platform to train complex strategic emergency response.

Using OLIVE platform, the research team created an Incident Command and Control Centre (Figure 1), and a simulated area—Willsdon Town in Silverdale County (Figure 2). These are based on the comments made by the advisory groups from the U.K. Category 1 responders.

Figure 1.

Snapshot of the command and control center of the exercise.

Figure 2.

An example of the flood risk map.

Existing licensed software such as ARGIS and Photoshop were used to produce geographic-information-simulated population data, vulnerability and risk indices, meteorological data and flood risk information for the area, for example (Figure 2). The characteristic of this data is as close to reality as possible; hence, the social vulnerability data is built by comparing the demographic trend in the United Kingdom, which includes gender, age, occupation, ethnic groups, income level, and disability in each ward.

In order to better express realism, topography for the region was developed from existing Digital Elevation Models, and associated flood risk maps, infrastructure and architecture for critical locations, population demographics, administrative boundaries, and emergency management resources across the urban environment will be designed. The simulated area is also demonstrated by using 3D Google Earth maps (Figure 3). Participants could find information in the Inventory.

Figure 3.

3D Google Earth map of Willsdon Town in Silverdale County.

Functions such as message board (to allow private and public communication or record keeping), voice (to allow public chat), radio channels (for private chat), and PowerPoint slide board (to disseminate injects and general information) were also included in-world.

The Progress of the Exercise

In order to complete the exercise, participants were required to take part in a 3-hour pre-exercise practice session, and the actual exercise. The exercise itself lasted about three 3 hours, which included a briefing, exercise (Games 1 and 2), and a debriefing session (see Table 1).

Table 1.

Timetable of the Exercise.

Time	Activities
Day 1: 10.00-13.00	Practice session
Day 2: 10.00-11.00	Game 1: Flood warning
Day 2: 11.00-11.30	Interim: Video/news
Day 2: 11.30-12.30	Game 2: Flood response
Day 2: 12.30-13.30	Debriefing

In the pre-exercise practice session, each participant was provided with an avatar, and learned and practiced relevant skills to operate their avatars. In the pre-exercise practice session, participants were introduced to the function of the platform. They were also provided with commands to operate the platform and avatars, for example, using radio channels and the inventory. It was believed that once participants were familiar with the platform, they could concentrate on the injects and the exercise itself on the real exercise day.

The exercise lasted about 3 hours, starting from a 10-minute briefing session to inform participants of the aims and objectives of the exercise. The procedure and timetable of the exercise were outlined. The house keeping rules of the exercise were also introduced to the participants. This was to prevent participants from violating the rules in virtual worlds.

The actual exercise started after the briefing session. The exercise was divided into two phases: flood warning and flood response phases. Game 1 (flood warning) lasted about 1 hour. Participants received a severe rainfall alert issued by the Environment Agency, which required participants’ attention for potential risks, including flood hazards. Following this, a severe flood warning was issued, and the “police” held a multiagency Strategic Co-ordinating Group (SCG) meeting at the command and control center to discuss issues relating to preparation for a flood emergency according to Emergency (Cabinet Office, 2012). This was the official exercise start state. The exercise uses the Met Office National Severe Weather Warning Service guidance document; an example of an extreme rainfall alert is shown in Figure 4.

Figure 4.

An example of the weather guidance document.

In opposition to traditional exercises that normally use several injects to create a chaotic and complex situation to represent reality, during exercise Game 1, each participant was provided with about two injects only: extreme rainfall alert and preparation for a potential flood risk. It was argued that participants would immerse in the exercise and gain more insights regarding multiagency coordination and responding to a major catastrophe if they could find a way around in the platform, understanding the meaning of the injects, and further make appropriate decisions and take action. At the end of Game 1, participants were required to produce a report to COBR (the Cabinet Office Briefing Rooms).

A 30-minute break was planned between Games 1 and 2. Before Game 2 started, a video demonstrating the development of the flood was presented to the participants. The plan was to use streamlining of the OLIVE platform; however, due to a limitation and technique failure during the exercise date, only a YouTube video clip was played. The video fitted well into the scenario. It illustrated the impact of the flood hazard, and the response of local communities. The use of video was to simulate the press release and the influence of media during a disastrous event.

Game 2 was the peak flood period; hence, the scenario became more complex. Participants were required to shift from preparedness to response where injects would develop their decision-making and aligned skills associated with the rescue and care of displaced public, and dealing with critical incidents associated with the flooding, including electricity breakdown, HazMat pollution event, and loss of water across the region. It is clear that in Game 2, more challenging injects were provided to the participants.

In Game 2, the research used an “unstructured” inject to facilitate five learning outcomes decided by the research team. Traditionally, emergency response exercises used structured injects; hence, participants would be guided to take certain action after they received an inject. The research team felt that with rigorous guidance, participants would learn rules and regulations well but fail to respond to crisis events. With unstructured injects, participants were provided with a “situation” but not with any hint of what to do; this is to train for the skills and competencies that are required for responding to crises and emergencies. Participants were required to investigate possible options and make decisions; they also needed to ask for information. The research team also prepared a supportive method and backup plan; if participants did not react as planned, they would be given hints on the walls (PowerPoint slides) and information stored in the inventory for participants’ reference. At the end of Game 2, participants were required again to produce a report to COBR.

Debriefing and Evaluation

In order to identify and reflect on learning outcomes, participants were invited to join a debriefing session immediately after the exercise was finished. At the beginning of the debriefing session, participants were asked to fill in a questionnaire for participants to assess their performance as well as to evaluate the effectiveness of the exercise and the platform. Questions included their performance in the exercise, the platform itself, immersion of using the game to train emergency responses, and injects.

In the debriefing session, participants answered a series of open-ended questions verbally and in written format regarding the scenario, story line, the use of the OLIVE platform, and the given learning outcomes, which included, (a) What went well in the exercises? (b) What difficulties did you find with the exercise? and (c) What could be improved in the exercise. A series of hot debriefing questions were also provided for participants to reflect on their performance and/or decision making.

Evaluation Framework

Training exercises involve much money and personnel. For example, they require a large number of participants, exercise controllers, observers, and training technologies, and validation becomes a central issue for emergency response training exercises (Lee et al., 2010; McCreight, 2011; Stolk et al., 2001). Validation refers to “the process of collecting, describing, recording, scoring and interpreting information about individual learners or groups” (Stretch, 2000, pp. 39-40).

Several frameworks have been developed to validate the design and study of training exercises. Raser (1996) points out four types of validity criteria, which include (a) psychological validity: the degree to which players accomplish the exercise—it also relates to learners’ needs and motivation; (b) structural validity: the degree of association of the simulated structure and the reference system; (c) process validity: the similarity between the process operating and developing in simulations and the reality; and (d) predictive validity: the degree to which a simulation could produce data or reference for future events.

Another example is the four dimensional framework proposed by de Freitas and Oliver (2006). They do not only focus on the structure and process of the learning environment, but also take pedagogy and effective learning outcomes into consideration (Chen et al., 2009; Rebolledo-Mendez, Burden, & de Freitas, 2008):

Dimension 1: Structure—Focuses on whether or not the stimulated structure represents the reference system. The research compares the representation of the organizational framework with the real world emergency management structure. In addition, the realism of the exercise is also examined, including the background of the scenario and the simulated city (geographic and demographic data and maps).

Dimension 2: Process—Refers to the development of the event that can be observed in the reference system. Hence, the development of each phase in the story line and timing of injects is examined.

Dimension 3: Context—Examines whether the exercise could be used in the context of learning and whether it has access to further learning resources. Usability of the platform is also one of the keys for evaluation, for example, whether it is easy to operate and easy to use at any time.

Dimension 4: Pedagogy—Considers whether the learning objective is provided to learners and the degree to which the exercise could be controlled to allow role-play and the exploration of competencies. Learners’ identity and interactivity in the exercise are also taken into consideration.

Based on the frameworks discussed, the research team determined the following criteria to evaluate the exercise: the representation of reality, context of the environment, immersion and usability, and learning outcomes.

Evaluation of the Exercise and Findings

The data regarding the evaluation of the exercise were based on the questionnaire after the exercise conducted in a university in England. In all, 11 participants joined the exercise, playing Category 1 strategic responders. Two of them played as exercise facilitators and one observer who was expert in designing and conducting exercises. Although the volume of data was small, it was hoped that more exercises could be conducted to increase the reliability and validity of the findings.

The Representation of Reality

Fidelity examines the degree to which the presentation and assimilation of the in-world environment relates to the real world experience. It indicates to a certain degree that the design of the in-world environment highly represented the reality. Although most of the participants found it different from traditional tabletop exercises (91%), more than half of them found the in-world space similar to traditional tabletop exercises (54%). The majority of the participants answered that they found the same synergy as they would in their own organizations’ emergency response set-up (54%).

Most participants found the YouTube videos played between Games 1 and 2 useful for understanding the scenario (67%), and most of them also found them realistic (89%). This correlates with the argument that using multiple training and learning materials facilitates the capacity of learners to relate the scenario to the real world, thus further increasing their learning outcomes. These results were illustrated in Table 2.

Table 2.

Results Regarding the Representation of Reality.

	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I felt it was the same as traditional tabletop exercises.	36	55	0	9	0
I found the space similar in the OLIVE platform exercise to the tabletop exercise.	0	45	0	45	9
I found the same synergy as I would in my own organization’s emergency response.	0	18	27	45	9
I found the video played before Game 2 useful for realizing the scenario.	11	0	22	56	11
I found the video played before Game 2 realistic.	0	0	11	56	33

Note. OLIVE = On-Line Interactive Virtual Environment.

Context of the Environment

In terms of context of exercise environment, the research team examined the function of the exercise, for example, 3D interactions, collaboration facilities, sharable user content, and ease of operation, such as learner control and usability.

Most of the participants found it easy to operate avatars (91%). In general, most participants found it easy to control the functions installed in the platform, such as message (90%), radio channels (54%), and voice (61%). As we had problems in uploading the PowerPoint slides, participants found it difficult to operate the PowerPoint slide board (80%). The PowerPoint slide board provided participants with up to date information about the development of the flood disaster.

A majority of participants did not find the background information sufficient (64%; also see discussion on “Process”). This could be the result of the low percentage of usage of the “handbook” provided in the “break out room” in the virtual “Command and Control Centre” (54%). Most participants “stayed” in the main command and control room. They did not have a chance to go to the separate discussion rooms where the handbooks were located. It is suggested that the handbook be located in the main command and control room to increase its usages. For results, see Table 3.

Table 3.

Results Regarding the Context of the Environment.

	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I had difficulty moving the avatar.	55	36	9	0	0
Did you find the following function easy to use?
a. Message	0	0	10	30	60
b. Radio channels	0	36	9	27	27
c. Voice	0	0	36	36	27
d. PowerPoint slides	30	50	20	0	0
I had enough background information to ensure my decision making was effective and appropriate.	0	64	9	27	0
I referred to the handbook displayed on the table in the small meeting rooms frequently.	27	18	0	36	18

Immersions and Usability

It was easy for most of the participants to find their way about the OLIVE platform (82%). Although most of them did not feel isolated in the OLIVE platform (73%), only few could relate themselves to their avatars (27%). Only 27% of the participants felt it real. However, they saw the virtual world as a space to share with other people (73%; see Table 4).

Table 4.

Results Regarding Immersion and Usability.

	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I can find my way around in the OLIVE platform.	0	0	18	55	27
I feel isolated in the OLIVE platform.	18	55	18	9	0
I could relate to my avatar.	0	18	55	27	0
I felt it was real.	0	36	36	27	0
I was sharing a space with other people in the virtual world.	0	9	18	55	18

Note. OLIVE = On-Line Interactive Virtual Environment.

In terms of usability, most participants found it easy to communicate with other people (64%); however, only 45% found it effective when it came to communication with their team members (45%). As the exercise was conducted in the same classroom, only 45% of participants had a chance to discuss these issues “in world” with other participants, while 72% of them talked to other participants “in person” to discuss issues related to the emergencies (see Table 5).

Table 5.

Results Regarding Usability.

	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I communicated well in the OLIVE platform exercise.	0	36	64	0
I communicated with my team members effectively in the OLIVE platform exercise.	18	36	45	0
I had a chance to discuss and make decisions with my team members in the OLIVE platform exercise.	9	45	45	0
I talked to other participants “in person” while playing the exercise.	0	27	36	36

Note. OLIVE = On-Line Interactive Virtual Environment.

Process

Process validation examines the development of each phase in the story line and the timing of providing injects. The research team looked into how participants associated themselves with the injects and the story lines provided. The exercise was divided into two games. Game 1 was the pre-flood scene: Participants were provided with a “flood watch” warning from the Environment Agency and an “extreme rainfall alert” from the Flood Forecasting Center. In Game 2, three injects were given to participants, including electricity loss, water supply, and HazMat.

It was considered that traditional structured injects did not encourage participants to think creatively or to be flexible, so unstructured injects were used. Participants were only provided with background information of the inject. Information that participants may wish to use in response to the inject was prepared, but the participants needed to ask to receive the information. More information was only given when the participants were really stuck for a response.

However, the results of the questionnaire show a negative feedback to the unstructured injects. Although participants found the injects appropriate (64%) in Game 1, most of them felt that they did not have sufficient information to help them make decisions (64%); only 45% of them thought that they made decisions effectively. One of the reasons could be that most of the participants in the exercise were novices to emergencies. It may be easier for the experienced professionals to utilize unstructured injects. Feedback from the exercise control regarding this point was that participants were looking for information for tactical commanders, rather than strategic managers. Consequently, it becomes a big challenge for them to seek appropriate information and think of potential decisions that could be made (see Table 6).

Table 6.

Results Regarding Process.

	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I had enough background information to ensure my decision making was effective and appropriate?	64	9	27	0
I felt the injects sent to my organization in Game 1 were appropriate for what would be sent in a situation where gold has been preemptively set up?	27	9	45	18
I made decisions effectively in the OLIVE platform exercise.	9	45	36	9

Note. OLIVE = On-Line Interactive Virtual Environment.

Learning Outcomes

The results of learning outcomes demonstrate a couple of controversies. Most of the participants agreed that they had learned something about the virtual environment (91%), but only a few learned the skills required to respond to emergencies (27%). When they compared the OLIVE platform with a traditional tabletop exercise, most of them agreed that they learned more on traditional exercises (91%). Most of the participants did not think they have done a good job in the exercise (82%). This could be a result of participants not being clear about the objectives of the exercise and the effects of the unstructured injects. It could also be related to the degree to which participants immersed themselves in-world, and the degree to which they were motivated to complete the missions during the exercise. It is also necessary to look into the qualitative data.

Although participants had passive attitudes toward learning outcomes, they found the OLIVE platform could be one of the desirable online learning environments for emergency response exercises and training (91%), and 45% of them felt strongly that the OLIVE platform has considerable potential to become one of the better methodologies to run emergency response exercises and training (see Table 7).

Table 7.

Results Regarding Learning Outcomes of the Exercise.

	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
I learnt something about the virtual spaces.	0	0	9	73	18
I learnt the skills that are required to respond to emergencies.	0	18	55	27	0
Comparison of OLIVE and traditional tabletop emergency response exercise: I find myself learning more effectively on the OLIVE platform.	27	64	0	9	0
Overall, as a strategic manager, I think I have done a good job.	9	73	18	0	0
I think the OLIVE platform exercise could be one of the online learning environments for emergency response exercising and training.	0	0	9	64	27
I think OLIVE platform exercise has the potential to become one of the methodologies to run emergency response exercises and training.	0	0	0	55	45

Note. OLIVE = On-Line Interactive Virtual Environment.

Overall, if this game is going to be utilized by professional practitioners, as first time players in an OLIVE emergency, it is suggested, first, that they be provided with relevant information as early as possible so they will be able to study the simulated area, and second, if any specific county/region decides to use OLIVE as their long-term training vehicle, then the platform can be easily adjusted to meet their individual requirements.

Discussion

Preliminary findings shows that with high levels of realism, the research enables strategic decision makers to come together to train and refine strategies more frequently and cost effectively than if they had to rely on traditional tabletop or live exercises. However, it is suggested that the realism should be related to the experience of the participants’ workplace. According to the arguments by Bramley (1991) and Lim et al. (2007), effective vocational training should take place in the actual working environment.

Although high percentage of participants found it difficult to immerse in-world, this platform enabled participants to adopt avatars within the environment that can closely resemble the role that they would be expected to play in the event of a flood emergency, although currently, this is only a small sample size.

With a high acceptance of the use of the OLIVE platform, the other advantage is that functions such as command and control centers, communication systems, external environment, and information resources can be adjusted to the requirements of individual organizations and geographical locations.

It can also be adapted to resemble both the dynamic environment of a command and control center, and the external flooded urban environment. Engaging in such exercises will allow participants to gain an in-depth and shared understanding of multiagency response mechanisms and associated strategic issues. The development of decision-making skills at the strategic-tactic level and the communication to the public enable government-mandated auditing of emergency planning, training, and exercising activities.

The next step is to explore and develop games targeted at enhancing communication and coordination skills between strategic, tactic, and the general public in order to respond to emergencies or disasters more effectively. In addition, developing several scenarios for emergency services at all levels has been identified by the research team as an area for future investigation.

Conclusion

This article described the theoretical background of the strategic flood emergency response exercise in the OLIVE platform designed by the research team. The article also described and explained how the exercise was designed and conducted. The article further summarized the evaluation of the exercise by analyzing a questionnaire completed by the participants. The results suggest, first, that a majority of participants were positive about the new format of the exercise. Most of them found it as a great potential alternative for future exercises. Second, the OLIVE exercise increased communication and coordination among participants. However, fidelity of the exercise has become crucial in the design and conduct of the exercise. Although literatures argue that high fidelity is not necessary, it is found that a crucial factor is to link the exercise to participants’ daily job environment. It was noted that participants could not relate the scenario to their learning activities as the scenario was based on a simulated area.

For future research to be more effective, it is suggested that collaboration between developers, academics, and the end users be encouraged. By doing so, the training exercise can then produce a scenario that suits better for users’ need and provide all parties with more accurate information. In addition, the collaboration will allow more opportunities to review and revise the exercise design and methodologies.

As most of the participants saw virtual worlds as alternative learning methodologies, it would be invaluable to develop a series of exercises, in particular, linking emergency responders at all levels in the United Kingdom, as they are the strategic, tactical, and operational responders. Pedagogically, by using consistent and continuous training with the same format, it meets the aim and goal of continuity personal development (CPD). The other opportunity when using virtual worlds is to provide decision makers with an alternative device to communicate and coordinate with other relevant personnel. As the research had a limited budget, it would be useful to conduct a larger scale evaluation to identify the usability of the exercise in a virtual environment.

Footnotes

Acknowledgements

The author would like to thank the valuable comments from the reviewers. The author also thanks the contribution of the research team, in particular, El Parker and James Townsend.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funded by the Advantage West Midlands, the research project was a collaborative undertaking by Ambient Performance and Coventry University.

Author Biography

Yung-Fang Chen (PhD) is a senior lecturer in disaster management and emergency planning at Coventry University, UK. The focus of her research is in the context of using training exercises to facilitate risk communication between public service providers and private companies in order to respond to catastrophic disasters more effectively. Her current projects include tasks for postdisaster assistance and reconstruction.

Contact: yung-fang.chen@coventry.ac.uk.

References

Alexander

(2000). Scenario methodology for teaching principles of emergency management. Disaster Prevention and Management, 9, 89-97.

Alexander

(2001). E-learning developments and experiences. Education + Training, 43, 240-248.

Asan

(2003). School experience course with multimedia in teacher education. Journal of Computer Assisted Learning, 19, 21-34.

Bartle

R. A.

(2006). Virtual worldliness. In Balkin

J. M.

Noveck

B. S.

(Eds.), State of play: Law, games, and virtual worlds (pp. 31-54). New York: New York University Press.

Biocca

(1997) The cyborg’s dilemma: Progressive embodiment in virtual environment. Journal of Computer-Medicated Communication, 3(2). Retrieved from http://jcmc.indiana.edu/vol3/issue2/biocca2.html.

Boin

Kofman-Bos

Overdijk

(2004). Crisis simulations: Exploring tomorrow’s vulnerabilities and threats. Simulation & Gaming, 35, 378-393.

Bramley

P. B.

(1991). Evaluating training effectiveness-translating theory into practice. London, England: McGraw-Hill.

Burrow

Berardinelli

(2003). Systematic performance improvement-refining the space between learning and results. Journal of Workplace learning, 15, 6-13.

Cabinet Office. (2012). Emergency response and recovery: Non statutory guidance accompanying the Civil Contingencies Act 2004. Retrieved from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/61047/Emergency-Response-Recovery-24-7-12.pdf

10.

Cabinet Office. (2013). Emergency planning and preparedness: Exercises and training: How to run exercises and training for emergency planning and preparedness, with an introduction to the Central Government Emergency Response Training (CGERT) Course. Retrieved from https://www.gov.uk/emergency-planning-and-preparedness-exercises-and-training

11.

Chen

Y. F.

Parker

Townsend

(2009, December). Exercise Atlantis: A virtual strategic incident management training (vSIM) for strategic emergency response. Taiwan Disaster Management Conference, Taipei, Taiwan.

12.

Chen

Y. F.

Rebolledo-Mendez

Liarokapis

de Freitas

Parker

(2008, November). Using On-Line Interactive Virtual Environment to develop training methods for emergency services. C-Games International Conference, Wolverhampton, UK.

13.

Cole

J. M.

Hilliard

V. R.

(2006). The effects of web-based reading curriculum on children’s reading performance and motivation. Journal of Educational Computing Research, 34, 353-380.

14.

Collis

(1996). Tele-learning in a digital world. London, England: Thomson Computer Press.

15.

Connolly

T. M.

Stansfield

M. H.

(2006). Using games-based eLearning technologies in overcoming difficulties in teaching information systems. Journal of IT Education, 5, 459-476.

16.

de Freitas

(2008). Serious virtual worlds: Scoping study. Bristol, UK: Joint Information Systems Committee.

17.

de Freitas

Oliver

(2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? [Special issue]. Computers & Education, 46, 249-264.

18.

deLeon

Killian

(2000). Comparing modes of delivery: Classroom and on-line (and other) learning. Journal of Public Affairs Education, 6, 5-18.

19.

Ford

J. K.

Schmidt

A. M.

(2000). Emergency response training: Strategies for enhancing real-world performance, Journal of Hazardous Materials, 75, 195-215.

20.

Forterra Systems. (2007). OLIVE-Purpose driven virtual worlds for everyone. Retrieved from http://www.forterrainc.com/images/stories/pdf/OLIVE_Dec07_Final_Rev.pdf

21.

Granlund

(2001). Web-based micro-world simulation for emergency management training. Future Generation Computer Systems, 17, 561-572.

22.

Green

W. G.

(2000). Exercise alternatives for training emergency management command center staffs. Boca Raton, FL: Universal Publisher.

23.

Gunasekaran

McNeil

R. D.

Shaul

(2002). E-learning: Research and applications. Industrial and Commercial Training, 34, 44-53.

24.

Halkett

(2002). E-learning and how to survive it. Industrial and Commercial Training, 34, 80-82.

25.

Higher Education Funding Council for England. (2005). HEFCE strategy for e-learning [online]. Retrieved from http://www.hefce.ac.uk/pubs/hefce/2005/05_12/05_12.pdf

26.

Lee

Trim

Upton

(2010). Large emergency-response exercises: Qualitative characteristics: A survey. Simulation & Gaming, 40, 726-751.

27.

Liarokapis

Macan

Malone

Rebolledo-Mendez

de Freitas

(2009). A Pervasive augmented reality serious dame. Conference in Games and Virtual Worlds for Serious Applications. VS-GAMES’09, 148-155.

28.

Lim

Lee

S. G.

Nam

(2007). Validating e-learning factors affecting training effectiveness. International Journal of Information Management, 27, 22-35.

29.

McCreight

(2011). An introduction to emergency exercise design and evaluation. Lanham, MD: Government Institutes.

30.

Michael

D. R.

Chen

S. L.

(2005). Serious Games: Games that educate, train, and inform. Boston, MA: Thomson Course Technology.

31.

Raser

J. R.

(1996). Simulation and society: An exploration of scientific gaming. Boston, MA: Allyn & Bacon.

32.

Rebolledo-Mendez

Burden

de Freitas

(2008). A model of motivation for virtual-worlds avatars. In Prendinger

Lester

Ishizuka

(Eds.), Intelligent virtual agents (Vol. 5208, pp. 535-536). Berlin, Germany: Springer.

33.

Robert

Gamelin

Hausler

Jarry

(1996). Training concept for environmental emergency measures structuring knowledge. Journal of Contingencies and Crisis Management, 4, 175-183.

34.

Schaafstal

A. M.

Johnston

J. H.

Oser

R. L.

(2001). Training teams for emergency management. Computers in Human Behaviour, 17, 615-626.

35.

Shen

Wang

Shen

(2009). Affective e-learning: Using “emotional” data to improve learning in pervasive learning environment. Educational Technology & Society, 12, 176-189.

36.

Stolk

Alexandrian

Gros

Paggio

(2001). Gaming and multimedia applications for environmental crisis management training. Computers in Human Behavior, 17, 627-642.

37.

Stretch

D. C.

(2000). Simulation design. In Saunders

Smalley

(Eds.), Simulations and games for transition and change. The international simulation and gaming research yearbook (Vol. 8, pp. 33-41). London: Kogan Page.

38.

Vector Command. (2010). Emergency command system: Command support system. Retrieved from http://www.emergencycommandsystem.com/products/command-support-system/

39.

Westera

(1999). Paradoxes in open, networked learning environments: Toward a paradigm shift. Educational Technology, 39, 17-23.

40.

Youngblood

Hedman

Creutzfeld

Fellander-Tsai

Stengard

Hansen

Heinrichs

W L.

(2007).Virtual worlds for teaching the new CPR to high school Students. Studies in Health Technologies and Informatics, 125, 515-519.