Learning Differences Between 3D vs. 2D Entertainment and Educational Games

Introduction

Many complex and abstract scientific concepts are taught in physics classes. In a traditional setting, a teacher presents explicit knowledge and formulas pertinent to the subject matter of physics. It can be difficult to explain abstract concepts such as Newtonian Laws of Motion (White, 1984), especially when the teacher needs to correct existing misconceptions (Eryilmaz, 2002, p. 1012). To correct misconceptions, a learner must adjust their intuitive cognitive concepts with a process known as cognitive change (McDermott, 1984, 1990; Scott, Asoko, & Leach, 2007). Experiential learning (D. A. Kolb, 1984; A. Y. Kolb & Kolb, 2009) and scientific discovery learning (D. A. Kolb, 1984; White, 1984) have proven to be successful approaches for correcting misconceptions that a learner may currently possess. The learner is confronted with a new experience that challenges the existing misconception. “Laboratory inquiry alone is not sufficient to enable students to construct the complex conceptual understandings of the contemporary scientific community.” (Lunetta, Hofstein, & Clough, 2007, p. 405). In a physics class, hand-on exercises are often used to provide the experience needed, and afterwards, together with the teacher, the learner reflects on the experience.

As computers have entered the classroom, electronic simulations have become available for scientific discovery learning. It is now possible to provide experiences that are not possible in real life, where the events can be structured in simulations. Learners can benefit from model progression and a level structure that help them to plan and structure the exploration (de Jong & van Joolingen, 1998). The combination of model progression and level structure inspired many researchers to design educational games that help learners grasp abstract concepts and instigate cognitive change. (Garris, Ahlers, & Driskell, 2002; Hoevenaar, 2010; Kearney & Pivec, 2007; Kiili, 2006; Koops & Hoevenaar, 2012; Leemkuil & de Jong, 2011; Paras & Bizzocchi, 2005; Rieber & Noah, 2008; Verheul & Dijk, 2009).

In an earlier study, Koops and Hoevenaar (2012) found that cognitive change for Newtonian motion can be realized by using SPACECHALLENGE, a game designed especially for this purpose. One drawback of this approach is that it is very expensive to design learning games for every single concept in school. To avoid high development costs, the use of Commercial-Off-The-Shelf (COTS) games was suggested (Sandford, Ulicsak, Facer, & Rudd, 2006; Verheul & Dijk, 2009). In this article, the use of a commercial three-dimensional (3D) entertainment game for the same purpose is investigated. Commercial entertainment games are known to facilitate learning (Dempsey, Haynes, Lucassen, & Casey, 2002; Egenfeldt-Nielsen, 2006; Verheul & van Dijk, 2009). The quality of the commercial entertainment games is what the learners expect from such games (Lampton, Bliss, & Morris, 2002; Pivec, 2008). Learners are accustomed to high quality entertainment games and might expect this kind of quality in educational games – this can result in disappointment and lowered motivation when these expectations are not met (Lampton et al., 2002; Pivec, 2008).

This article describes an investigation to ascertain whether a commercial 3D game can yield a higher learning gain than a similar two-dimensional (2D) educational game. The main goal of using a game in physics lessons is to provide the learners with an experience. It is important that the game behaves according to the correct physics laws (de Jong & van Joolingen, 1998) to achieve the learning objectives. Depending on the learning goal, different key aspects (cues) can be emphasized (Visschedijk, 2010). For example, Limniou (2008) outlined how a 3D simulation of chemical reactions leads to greater learning gain than a 2D simulation of the same subject. When it comes to the requirement for a simulation game on Newtonian laws, the best representation is unknown. This study compares the highly realistic immersive commercial game and a simple, self-designed 2-dimensional educational game.

An important aspect in learning is flow (Csikszentmihalyi, 1988). Unlike computer simulations or applets that are used in the classroom, entertainment computer games are likely to provide an immersive experience and trigger a flow state. Many researchers share the conviction that the flow state is one of the conditions for effective, deep, learning (Castell & Jenson, 2003; Hamari et al., 2016; Kearney & Pivec, 2007; Paras & Bizzocchi, 2005; Scoresby & Shelton, 2011). This deep learning will take place on a tacit level. The player intuitively reflects on the experience and decides on new actions, on the flow (Garris et al., 2002; Rowe, Asbell-Clarke, Bardar, Kasman, & MacEachern, 2014). However, it was also observed that highly realistic games do not necessarily yield the best training results (Feinstein & Cannon, 2002; Mania, Troscianko, Hawkes, & Chalmers, 2003). For Appelman (2007), “the confusion in a serious game is whether or not the flow is synonymous with engagement in learning or just deep concentration on gameplay and fun” (p. 1). In their study, Rieber and Noah (2008) found that “the gaming activity did not promote reflective cognition. In fact it actually interfered with explicit learning” (p. 88).

When comparing a 3D commercial game to a 2D educational game, we wanted to determine whether a difference in flow experience occurs. We expected that a commercial 3-dimensional entertainment game would be more engaging and immersive than a self-designed 2-dimensional game. Immersion is often described as a sense of being present and a 3D virtual environment is likely to increase immersion (Cairns, Cox, Berthouze, Dhoparee, & Jennett et al., 2008). The commercial game, UNREAL TOURNAMENT) was used in this study along with a self-designed 2D game, developed by a teacher using the free GameMaker engine.

We previously developed a 2D educational game SPACECHALLENGE (Koops & Hoevenaar, 2012), and utilised the game to teach a very specific learning goal, that is, conceptual change concerning Newtonian motion. In this study, a commercial entertainment game, UNREAL, was utilized to mimic the SPACECHALLENGE behavior in a multiplayer, 3D mode. For the control condition, a set of hands-on experiments were designed to induce cognitive change concerning Newtonian motion. To test our hypotheses and find out which of the three approaches generates the largest cognitive change regarding the concepts of Newtonian motion, the learning gain of a lesson series containing one of the three experiences was measured: learning game, entertainment game or hands-on experiment,. Additionally, the experience of flow was also measured.

Requirements

To induce conceptual change, the following conditions need to be met: (i) the learners must first be dissatisfied with existing ideas, and (ii) new ideas must evolve that are perceived as intelligible, plausible and fruitful (Posner, Strike, Hewson, & Gertzog, 1982). One of the suggestions Posner et al. (1982) made as a teaching strategy was to develop activities that create cognitive conflicts in learners. Furthermore, it is important that the teacher discusses the misconceptions and cognitive conflicts with the learner, using multiple representations (verbal, mathematical and practical).

Some form of action needs to be designed that gives rise to a cognitive conflict, in order for the learner to feel there is a need for cognitive change. The learner needs to be supported in adopting a new scientific concept that is intelligible, plausible and fruitful. We represented these requirements in the Serious Gaming Lemniscate Model (SGLM) (Koops, 2010). The SGLM is presented in Figure 1. It represents a Kolb learning cycle, with the option to replace the concrete experience with a game cycle. When applied to conceptual change, the game cycle is used to create a cognitive conflict. In the game cycle, the learners are provided with a clear goal and an intuitive interface, where they decide on an action based on the existing conception. The game reacts and provides feedback to the learners, who can observe whether it is according to their expectation or not. It is the teacher’s task to accompany the learners in the learning cycle to discuss the observed phenomena. Here, the teacher teaches scientific concepts, proving them plausible and intelligent. During subsequent experiences, the learners enters the game cycle to apply the scientific concepts and experience their consequences. We will now describe the order of events that led to conceptual change in our research as it relates to Newtonian motion.

OPEN IN VIEWER

Figure 1.

The Serious Gaming Lemniscate Model. Source. Adapted from (Kolb &Kolb, 2009).

In the first lessons, the physical quantities of force, velocity and acceleration were introduced, alongside pertinent scientific means to record these. When the learners became familiar with these concepts, they were then introduced to the game or hands-on experiments, providing the learners with a learning experience. After the game, the relation between force, motion and acceleration were reflected upon and the formal Newtonian laws of motion were introduced. Finally, the processing task concludes the events where the students are asked to make predictions about motion, based on scenarios from the game or hands-on experiments.

The experience is presented in a similar way for the games and the hands-on experiments. The scenarios grow in complexity, beginning with a simple, linear motion where the learner must start (accelerate) and stop (decelerate). The assignments grow in complexity and curved and narrow paths are introduced.

In this research, three conditions were created, differing in the activities that took place in the game-cycle. In the learning cycle, the same lessons for each condition were provided. These include:

The hypotheses were tested by comparing the learning gain and the flow experienced by the Flow was measured using the Flow Scale for Games (Kiili, 2006). The learning gain, defined as the difference between pre-test and post-test scores, was measured by comparing the scores on an adapted version of the Force Concept Inventory (Hestenes, Wells, & Swackhamer, 1992).

The Games

The cognitive change that we aimed to reach is the correction in the misconception that a force is needed for motion. Newton discovered that an object, when no net force is applied to it, moves at constant velocity. The only way to change the velocity of an object, to accelerate or decelerate it, is to exert a force on it. Many students have the misconception that an object, when no force is applied, will eventually come to rest. Of course this is true on earth, in everyday life, where friction is always present. Consequently, it is almost impossible to provide a real life experience that confirms Newton’s law. In the virtual game world, we can turn off friction easily, creating an environment with new experiences. In a previous study, Koops and Hoevenaar (2012) created such an environment with the 2D game, SPACECHALLENGE.

SPACECHALLENGE

SPACECHALLENGE is a simple 2 dimensional, single player, simulation-game in which a player has to move a spaceship through a maze, collecting diamonds on the way. The game was created in GameMaker, a free available game editor, and proved to be able to induce the conceptual change. Students who played the game showed an increased understanding of the concepts of force and motion (Koops & Hoevenaar, 2012). The game consists of five different levels, increasing in difficulty. The level design was created by a physics teacher. In the first level, the students become familiar with the interface; the spaceship must move along a straight line and stop at the stop-sign. Motion is initiated using the arrow keys. Flames and a thrust-sound indicate whether the engine of the spaceship is active. No friction is present. The second level is identical, except for a blue background and a warning that air friction is present.

The 3^rd and 4^th levels are similar, but more complex. The 5th level is shown in Figure 2. The lower left corner shows a map of the complete maze. It becomes narrower as the player progresses. When three errors are made, the game is over and the player has to start over again. Health and fuel can be upgraded at specific load-points. The player must ‘hold still’ to benefit from them. The task of holding still is difficult without any friction and it tests whether the player understands the concept. Understanding the concept is not enough though – the concept must be executed in the motion of the spaceships.

OPEN IN VIEWER

Figure 2.

Level 5 SPACECHALLENGE.

UNREAL

The authors looked for an entertainment game that could be used to offer the same experience of frictionless motion to the students based on the idea that learning might be deeper when students are in a deep flow state (Castell & Jenson, 2003; Garris et al., 2002; Kearney & Pivec, 2007; Paras & Bizzocchi, 2005). The commercial 3D, first person shooter game, UNREAL TOURNAMENT was deemed suitable. The game provides 3D virtual worlds to walk and fly around in, with thirty players at the same time. The game is accompanied by a level-editor that allowed the creators to modify the game to their specific needs and change the gameplay in such a way that it mimics the gameplay from SPACECHALLENGE. Friction was turned off, and weapons were removed from the game. The game was rated as 16+, while our participants include 14 and 15 years old. A violence-free version of the game was created where the players collect coins from little treasure boxes that are placed throughout the world. This is very similar to the diamonds that are collected in the educational game SPACECHALLENGE.

The game goal is essentially identical to the goal in SPACECHALLENGE: collect items, maneuvre in a frictionless world, and sometimes hold still. The coins have to be delivered at specific drop points. This is comparable to the spaceship in SPACECHALLENGE that needs to remain stationary to re-fuel. During the game, players can bump into one another and steal coins from each other (Figure 3). The player must move over a treasure box to collect the coins from it. The collected coins must be delivered to the droppoint, indicated by the white moving circles. In the first level, friction is present and players can become familiar with the interface and the rockets. Subsequent levels lead to different worlds, albeit without friction. The player maneuvers through the environment using the computer arrowkeys.

OPEN IN VIEWER

Figure 3.

First Level UNREAL Mod.

Methodology

For this study, we compared three groups of students, who received the same lesson series, only differing in the game that was played. One group played a commercial 3D game (UNREAL-group), one group played a 2D educational game (SPACE-group), and a third group (control group) completed hands-on experiments, but didn’t play any game.

The study was undertaken at Unic, a secondary school in Utrecht, the Netherlands. All 3rd graders (age 15) took part in the study. Table 1 indicates the gender and sizes of each group.

OPEN IN VIEWER

Table 1.

Subjects by Group and Gender.

Group	Gender		Total
	Boys	Girls
UNREAL	12	18	30
SPACE	9	21	30
Control	16	17	33
Total	37	56	93

Before beginning the research, we conducted statistical tests on the students’ physics grades to ascertain if differences existed. This was not the case. Therefore, the authors had reason to believe that the division in the groups was random, pertaining to their knowledge of physics.

Not all students were always present, so the number of students available for the final analysis is slightly lower. The study was completed over a period of two weeks. In the first week, the research was introduced to all 93 students, while the processes of the research were also detailed. The goal of the research and the research plan were not explained to the participants. Immediately after this first-session, the understanding of the concept of force and the student’s motivation were measured. Three students, one from each group were interviewed. During the rest of the week, students worked on assignments on motion and acceleration. Students tended to work in teams. In the Unic system, plenary frontal lessons by a teacher are rare. A lesson is provided, content is available online, and the teacher coaches the teams.

At the end of the first week the game sessions took place. All three groups were simultaneously presented one of the games or the hands-on experiments in a 1-hour lesson. This maintained consistency in terms of the location of the game in the curriculum, and also prevented communication between the groups. Consequently, the teacher could not coach all three groups. Three independent observers guarded the game sessions. They did not help with any instruction and only managed the technical process. The students in the experimental game conditions filled in the flow scale immediately after the game-sessions were complete. In the week following the game session, the students, again simultaneously, completed a processing task. The tasks were identical for all groups, yet the illustrations were taken from the game-worlds (or the hands-on handouts) they had played in. These tasks were implemented as there was no opportunity for a debriefing on the game experience in the Unic system. The second week ended with filling in the post-tests, consisting of a questionnaire on understanding force concept and a questionnaire addressing motivation. Several evaluation questions pertaining to playing a game during school hours were asked.

The Processing Task

With no opportunity to do a proper debriefing, an individual, evaluation activity was conducted. Therefore, the experience concluded with a processing task instead of a verbal debriefing. There were no experiences or reflections exchanged between students. As the different groups gained experience in different settings, the images used for the processing tasks are also different (Figure 4).

OPEN IN VIEWER

Figure 4.

SPACECHALLENGE group processing question.

UNREAL Processing Task Example

Figure 5 depicts a top view of your avatar. Your avatar stands still at point A. In which order are the actions, indicated in the lower pane, performed in order to make it follow the trajectory that is indicated by the yellow line? (A dot means no action, an arrow indicates the direction of the force that is applied).

OPEN IN VIEWER

Figure 5.

UNREAL group processing question.

Hands-on Group Processing Task Example

In Figure 6 you see a drawing of the sheet of paper on which you have blown a ping-pong ball around. Below you see a series of actions (directions in which the straw is aimed towards the ball).

OPEN IN VIEWER

Figure 6.

Control group processing question.

What is the correct order of actions in order to make the ping-pong ball follow the trajectory indicated in the figure below?

Results

Conceptual Change

The scores of the FCI pretest were analyzed to determine the reliability of the modified FCI. The FCI consists of nineteen questions, and the maximum score is 19. We found α = 0.37 in the pretest (n = 92). This is a low reliability, which can be due to the fact that the scores on the FCI were fairly low. This reduces variance between the groups. The FCI scores are represented in Table 2 and Figure 7. From Figure 7 it is clear that the learning gain is largest for the SPACECHALLENGE group, and minimal for the UNREAL group.

OPEN IN VIEWER

Table 2.

FCI Averages and Standard Deviations by Group.

Group		Average	SD	N
Fci pretest	Practicum	5.3	1.9	26
	UNREAL	6.1	2.7	21
	SPACECHALLENGE	3.6	1.7	20
Fci posttest	Practicum	6.4	2.6	26
	UNREAL	6.1	3.6	21
	SPACECHALLENGE	5.6	1.7	20

Note. Significant gains between groups in a Games-Howell post-hoc test.

OPEN IN VIEWER

Figure 7.

FCI scores by group.

The highest score for the pre-test was 13, the highest score on the post-test was 14. Both of these results were recorded in the UNREAL group. A one-way ANOVA was used to analyze the test scores (=post-test score – pre-test score) among the three groups. The learning gain differed significantly across the three groups, F (2.64)  = 3.99, p = 0.023.

The effect size ω² is calculated for the ANOVA, and equals 0.08, a very small effect. Two contrasts were calculated. Contrast 1 is between the control group and both the game groups, it yields ω² = 0.03. Contrast 2 is between the Space group and the UNREAL group: ω² = 0.33, which is a quite a reasonable effect. One conclusion is that the games may not lead to a greater learning gain than hands-on experiments. However, educational games may be significantly more effective than other forms of games for physics instruction.

Flow

The results from the flow questionnaire are presented in Table 3. No significant difference in flow experience existed between the groups.

OPEN IN VIEWER

Table 3.

Flow Scale and Subscale Averages and Standard Deviations.

Group		Average	SD	N
Flow Total	UNREALSPACECHALLENGE	3.13.2	.081.0	2119
Flow Antecedents	UNREALSPACECHALLENGE	3.23.3	.081.3	2425
Flow Experience	UNREALSPACECHALLENGETotal	3.13.13.1	.08.09.09	222143

Discussion

The educational game induces the larger learning gain compared to the commercial game. This assertion stems from a comparison of the FCI scores of the post-test with those of the pretest. It is important to note that the game-intervention was only a small part of the lesson series, so the learning gain was measured where the game was the only differing factor. The group with the largest learning gain is also the group with the smallest FCI score to start with (Figure 7). One could argue that some kind of ceiling effect might limit the learning gain of the commercial game group. Groups with lower starting scores may also gain the most. It is also important to note that the theoretical maximum score for the FCI is 19. We measured the highest score (14) in the UNREAL group. In our previous research, (Koops & Hoevenaar, 2012), we observed that it was a lesson series containing SPACECHALLENGE that induced a larger learning gain than a lesson series that did not include other forms of gaming or a hands-on experiment. In that study, the SPACECHALLENGE group had the higher FCI scores to start with and also showed the highest learning gain.

The subject of conceptual change in this research is Newton’s 2^nd law. It is about motion, and consequently the motion in the games is the key factor. Although the experience in the commercial 3D first person shooter game is more realistic than the experience in the 2D educational game, there are many advantages in the latter. In the educational game, the player observes the motion from a third person top view, where the direction of the applied force is indicated by the spaceship’s flames. The player can form a mental model to describe what happens in the game. In the 3D perspective the direction of the force is not as clearly oriented in the environment. The learner must intuitively feel the forces and interpret the consequences. The representation is just as complicated as the real world and extracting the rules of motion from the experience is complex. The 3D perspective adds to the reality of the experience, but it does not add to the clarity of the motion’s nature. For our learning goal, the motion’s nature was a key factor and the 3D perspective did not add value.

It is remarkable that in the interviews, the students indicated that they did not learn much. It supports the idea that a game triggers tacit learning; the student is not aware of the learning, although there is significant influence of the game in the learning gain. While it is known that implicit knowledge increases more than explicit knowledge (de Jong & van Joolingen, 1998), the samples for the interview are basically too small to draw any conclusions.

The reported flow state for both types of games are equal. From the interviews and observation, it is apparent that the UNREAL group, who played the commercial game, seemed more enthusiastic and had more fun. The results of the questionnaire did not match the observations. Unfortunately the researchers did not investigate the flow experience of the control group. The control group was only examined to determine if the use of games did not have any negative effects on the learning gain.

Due to the school setting in this research, it was not possible to perform a teacher-led debriefing. The use of the processing task, after the game experiences, can be viewed as a short-cut in the SGLM in Figure 1. In this case, the learner experiences the nature of frictionless motion, and immediately starts a new active experimentation phase, doing the processing task. The reflective observation and abstract thinking are left for the learner to do by herself. One finding from this research is that the lessons can be improved when a debriefing session is added to include these phases in the learning process. In this session, the teacher focuses on the changed insights and invites the students to reflect and make these explicit. Students for instance can reflect on the FCI tests and explain the differences between pre- and post-test. This articulated reflection provides a good base for the teacher to enter the abstract conceptualization phase in the learning cycle.

Conclusion

In conclusion, students show a greater learning gain from playing the 2D educational game than the more immersive commercially developed UNREAL game. The students learn significantly more in a lesson series that contains the educational game. The players of both games detailed similar flow-experiences. This study shows that it can be realized without expensive commercial efforts.

The fact that the FCI scores are low overall might be because no debriefing was carried out. The Unic school model is based on students who work individually, where debriefing is the responsibility of the student. When applying educational games this can be a risky approach; a student who has met the game goals may be convinced that consequently, the learning goals were met as well. When it comes to conceptual change, the reflection upon the experience is crucial in changing existing misconceptions. Future research will focus on the integration of the debriefing process in the game itself.