The student experience of student-centered learning methods: Comparing gamification and flipped classroom

Abstract

Higher education enrollment has declined recently. New recruiting strategies and student-centered learning environments with challenge, enthusiasm, and joy are required to boost enrollment and engagement. Thus, the student-centered teaching approach, which one of the approaches under implementation and observation, are increasingly needed. Gamification and flipped classroom are cutting-edge educational approaches to motivate students. Nevertheless, the effectiveness of both methods has not yet been explored. The aim of this experimental study is thus to investigate and compare the effectiveness of these approaches. Student-led activities are designed and the perspectives from both participants and activity conductors are collected. The results show that these techniques successfully shift students’ perceived usefulness and engagement intention. Gamification gains slightly better results than flipped classroom from the participants’ viewpoints.

Keywords

Class activities assignments higher education active learning adoption information systems experimental teaching student surveys social science research

1. Introduction

University enrollment such as in Management Information Systems (MIS) has declined across the world (Buhl & Lehnert, 2012; Marshall et al., 2014; Rouibah, 2012) opposite to the growth of MIS and Information Technology (IT) in the job market (Rouibah, 2012). Thus, recruiting strategies are needed, together with educational activities as attractive motivators (Kanagasingam, 2017). Common approaches are redesigning curricula, transforming courses, and introducing new teaching approaches, particularly for the introductory courses (Kenny et al., 2017; Marshall et al., 2014), changing from lecture-based to self-directed learning (Kenny et al., 2017). Students are currently demotivated (Alabbasi, 2017). They need course enjoyment (Rouibah, 2012). Student-teacher interactions and student-centered learning environments together with teamwork concept could enhance their learning (Alabbasi, 2017; Schultz et al., 2014). Cutting-edge pedagogical techniques such as gamification and flipped classroom could be applied (Dolgopolovas et al., 2014; Moore, 2016; Virkus, 2015).

Gamification is the use of game elements in non-game context to increase their motivations (Kuo & Chuang, 2016; Sillaots, 2015). It has a potential to apply both in the commercial and educational market to change target audience’s behavior (Kanagasingam, 2017; Kuo & Chuang, 2016). Educational usefulness of gamification has been supported by the study of academic researchers who regularly apply the game mechanisms in their teaching process (Kanagasingam, 2017). Gamification promotes fun from learning by integrating more playful components (Kenny et al., 2017; Kuo & Chuang, 2016). It has been increasingly implemented lately to enhance the engagement among students (Aleksic-Maslac et al., 2017; Kanagasingam, 2017; Sillaots, 2015). It could also strengthen the students’ desire to learn more, increase their competitiveness, and motivate them for future development (Aleksic-Maslac et al., 2017). It augments student experiences and increase their responsibility to direct their learning (Armier Jr, et al., 2016; Llorens-Largo et al., 2016; Tan & Hew, 2016). However, the designers of gamification face constraints regarding a limited time and budget. They have to put more efforts and are quite difficult to analyze the detailed needs of each student as well (Kuo & Chuang, 2016). In addition, lecturers specify the main reason why they don’t use gamification because they are not familiar with gamification tools (Aleksic-Maslac et al., 2017). Flipped classroom moves away from an instructor-centered to student-centered learning (Dolgopolovas et al., 2014; Llorens-Largo et al., 2016), placing the lectures to be under the control of students using prerecorded readings or other technology-based materials (Adnan, 2017; Clark et al., 2016; Frania, 2014; Hopkins et al., 2017). Activities to practice with concepts are moved inside the classroom (Adnan, 2017). It also support peer-assisted learning, active learning, group work, and so on (Adnan, 2017; Moore, 2016).

Although some studies have investigated these two methods, as described in Section 2.2, gamification is quite difficult to implement (Cheong et al., 2013; losup & Epema, 2014; Veltsos, 2017). Gamification use in the future looks promising if it continues to create new solutions and motivations such as comparing individual engagement or student group engagement are analyzed (Aleksic-Maslac et al., 2017; Kanagasingam, 2017). Some professors feel unfamiliar with it (Aleksic-Maslac et al., 2017). Limited research about gamification exist (Armier Jr. et al., 2016; Hung, 2017; Kenny et al., 2017). Regarding the flipped classroom, there are some critics that it could lead to the less structured of the classroom dynamics (Hung, 2017). Thus, how best to conduct an effective flipped classroom should be explored (Adnan, 2017; Birkenkrahe & Kjellin, 2015; Hung, 2017). No research to date has investigated the effectiveness of these approaches using factors based on the Technology Acceptance Model (TAM) and the Expectation-Confirmation Model (ECM), to gain a deeper understanding of how student-centered learning methods impact students’ perceptions of TAM and ECM factors and how both methods differently influence their perceptions.

Therefore, the purposes of this study are to explore impacts of gamification and flipped classroom on students’ perceived usefulness, their engagement intention, and their satisfaction and to compare different effects of two student-centered learning methods, both in the aspects of participants and conductors. The goal of this study is to answer six research questions that are: RQ1) Do student-led gamification positively impact perceived usefulness and engagement intention of game participants? RQ2) Do student-led gamification promote perceived usefulness and engagement intention of game conductors? RQ3) Do student-led flipped classroom positively impact perceived usefulness and engagement intention of flipped classroom participants? RQ4) Do student-led flipped classroom promote perceived usefulness and engagement intention of flipped classroom conductors? RQ5) Are there any differences between perceived usefulness, engagement intention, and satisfaction of game/flipped classroom participants? and RQ6) Are there any differences between perceived usefulness and engagement intention of game/flipped classroom conductors?

2. Related theories and studies

2.1 The technology acceptance model (TAM) and expectation-confirmation model (ECM)

Main constructs that are explored in this study come from two main theories: the Technology Acceptance Model (TAM) and the Expectation-Confirmation Model (ECM). In general, these theories are applied in the IS/IT context. However, since this study applies student-centered learning methods with the MIS course. Also, games and flipped classrooms in this course normally utilize information technologies such as Kahoot or PowerPoint as supporting tools. It is interesting to utilize these established theories in different contexts to extend literature in education. In this study, perceived usefulness and satisfaction are deeply explored in various aspects, whereas behavioral intention/continuance intention is presented as engagement intention.

The TAM is an established theory in the IS field, which is extended from the theory of reasoned action (TRA) (Lee, 2010). It has proven to be a robust model to explain individuals’ behavior in the adoption phase (Orzan et al., 2012). Perceived usefulness is one of the key factors affecting an individual’s behavioral intention (Lee, 2010). Usage behavior is predominately explained by the behavioral intention as a result of decision-making processes (Shroff et al., 2011). Perceived usefulness refers to a user’s belief that the technology will improve his/her performance (Davis, 1989). The TAM is also applied in the educational context such as e-learning systems, online courses, or e-portfolio system (Liu et al., 2010; Shroff et al., 2011). In an online university context, perceived usefulness is related to the quality of the content (Joo et al., 2011). In other contexts, the TAM is also applied to explain online consumers’ intention to buy online or e-commerce websites usage (Orzan et al., 2012). However, solely TAM cannot represent a complete view of behavior beyond the adoption phase, which can be followed by continuance or discontinuance decision (Orzan et al., 2012).

The ECM posits that an individual’s intention to continue usage depends on three variables: the user’s level of satisfaction, the extent of user’s confirmation of expectations, and post-adoption expectations (Lee, 2010). The theory thus can be applied to examine the post-adoption phase and continuance intention (Orzan et al., 2012). It is used to explore how students’ confirmation of expectation about e-learning effects on their post-adoption expectation and how both of them influence satisfaction and continuance intention (Chow & Shi, 2014). It is also applied to explain consumers’ satisfaction and repurchase decisions in wide variety of contexts (Orzan et al., 2012). Post-adoption expectations means the degree to which users obtained their expected benefits through using a system (Chow & Shi, 2014). Therefore, it could be represented by perceived usefulness in some studies. In the Expectation-Confirmation Model of Continued IT Usage (ECM-IT) theory, which is developed from the combination of TAM and ECM, it also uses perceived usefulness to represent post-adoption expectations as well (Hong et al., 2005; Orzan et al., 2012). Based on the ECM, users will have a high level of satisfaction and continuance intention if they have a high level perception of their post-adoption expectations (Chow & Shi, 2014). Studies in marketing also point that satisfaction is a major reason for consumer’s decision to repurchase products or patronize services (Lee, 2010; Orzan et al., 2012).

2.2 Related literature

Chow and Shi (2014) explored the the expectation-confirmation model (ECM) factor of post-adoption expectation such as perceived usefulness that can be explained by post-adoption experiences of students using e-learning. Findings pointed that students’ confirmation of e-learning usage had a direct influence on quality assurance factors: learning process, tutor interaction, peer interaction, and course design. Dolgopolovas et al. (2014) studied the influence of students’ using self and time management tool on their attitudes toward the inverted classroom. Results showed that there was the positive shifts of student attitudes toward the inverted CS2 and the effectiveness of self and time management tool. Ferreira (2014) introduced a solution that utilizes the MOOCs (Massive Online Open Courses) to flip a classroom. The organization of the course, how to implement the solution and to assess the course’s efficiency were described. Mesquita et al., (2014) proposed the use of social networks and gamification mechanics to expand students’ participation. Most students performed better in the proposed educational environment due to its comfortable, fun, and communicative nature. Poole et al., (2014) investigated utilizing gamification elements and customer engagement principles in college business courses to foster Generation Y learners in learning. The findings supported the gamification as an effective tool to expand student engagement and learning.

Sillaots (2015) conducted a case study to examine how students perceived the gamified Computer Game course. Results showed the well acceptance of game elements by student, but it did not produce deep immerson to the course. Estévez-Ayres et al. (2015) proposed a methodology to facilitate the problem detection and reaction in active learning in engineering education. The use of active learning could improve the traditional classroom by promoting participation and engagement of students. Lameras and Moumoutzis (2015) presented the GamifyMaths framework integrating inquiry-based learning, the flipped classroom model, and gamification to improve the mathematics teaching. The Personal Learning Environment (PLE) offered engaging personalized learning experiences and promoted learners’ ownership, accomplishment, and creativity. Birkenkrahe and Kjellin (2015) presented the results from two experiments of flipped-graduate courses at the University of Stockholm and at the Berlin School of Economics and Law. Findings suggested that a design science approach could be applied to develop flipped classroom courses with some customizations using ongoing feedbacks from students. Latulipe et al., (2015) introduced lightweight teams, which included peer teaching, peer learning and long-term socialization, to make first programming classes socializable and effective for students. Lightweight teams combining with the flipped approach and gamification led to the increase of student engagement.

Tan and Hew (2016) explored how meaningful gamification using points, badges, and a leaderboard, impacted student learning, engagement, and affective outcomes in a blended learning research method class. Results indicated that the experimental group posted messages in the discussion forums and found the course motivated more than students in the control group. Clark et al. (2016) implemented the flipped classroom approach in senior engineering classes to drive active learning and student engagement. Findings pointed out that the flipped instruction positively affected the classroom atmosphere in terms of student discussion and questions. Armier Jr. et al. (2016) examined students’ willingness to participate in gamified activities, which reward systems were not directly linked to the course grades. Results showed the significant differences of individual participation, group pursuits, individual work, frequency of group meetings, group preparation, and purchased items between mid- and post-questionnaires and the significant differences of hours spent in groups between the treatment and the control groups. Kuo and Chuang (2016) applied gamification to an online platform for academic promotion and dissemination. Gamification had a power to attract, motivate, engage, and retain users. Hamari et al. (2016) studied the impact of flow, engagement, and immersion of games on learning. Results showed that engagement and challenge in the game positively affected learning, but immersion in the game did not significantly affect learning.

Aleksic-Maslac et al. (2017) showed an example of using Kahoot in Zagreb School of Economics and Management and investigated perspectives of professors and students regarding gamification. The findings showed the positive attitudes toward the use of gamification in the educational process. Hung (2017) aimed to improve the face-to-face instruction in the flipped classroom using a game-like clicker and Bring Your Own Device (BYOD) model. Results from the quasi-experiment indicated that the use of clickers had a positive impact on student learning in terms of students’ performance, perceptions, and preferences. Adnan (2017) integrated the flipped classroom approach into a senior-level course, adopting the action research methodology. There was no significant difference between the mean of midterm and final scores of the flipped and non-flipped groups. However, flipped students significantly gained higher essay scores more than the non-flipped students. Yildirim (2017) explored the effects of gamification-based teaching on student achievements and attitudes toward lessons, using quantitative research with a true experimental design. Findings revealed the positive impact of gamification-based practices on students’ achievements and their attitudes toward lessons. Alabbasi (2017) examined graduate students’ perspectives toward adding gamification to online learning. Graduate students showed their positive perceptions toward the use of gamification in online learning. Smith (2017) applied a quasi-experimental study to investigate the influence of gamified modules in a statistics course on students’ learning and attitudes toward statistics. Findings revealed that attitudes of cognitive competence, affection, value, and perceived difficulty were positively shifted after students completing the gamified exercises. The performance of students in the experiment group also increased after finishing the course.

3. Research hypotheses

There are three main hypotheses related to perceived usefulness, engagement intention, and satisfaction, which are defined in The Technology Acceptance Model (TAM) and Expectation-Confirmation Model (ECM) section. Hypotheses regarding perceived usefulness and engagement intention contain six sub-hypotheses with sub-components each, whereas there is one sub-hypothesis with sub-components about satisfaction.

3.1 Perceived usefulness

There is a strong influence of perceived usefulness on user’s intention of IT domain (Hong et al., 2005). Perceived usefulness of e-learning significantly positively affects users’ satisfaction and their continued usage intention (Lee, 2010). Post-purchase perceived usefulness significantly affects attitude towards online buying, which significantly influence consumers’ continued intention to buy online (Orzan et al., 2012). Perceived usefulness also a positive driver of the intention to use online learning community (Liu et al., 2010). Perceived usefulness significantly have positive impacts on learner satisfaction. The relationship between perceived usefulness and satisfaction of learners has been confirmed by several researchers (Joo et al., 2011). Post-adoption expectation (learning process, and course design) significantly influence e-learning satisfaction and continuance intention (Chow & Shi, 2014). Educational usefulness of gamification has been supported (Kanagasingam, 2017). Perceived usefulness affects intention to engage in gamification (Yang et al., 2017) and the continued use of gamification services (Hamari & Koivisto, 2015). Flipping engineering courses provide various benefits that are enhanced learning process, engagement, and so on (Clark et al., 2016).

3.1.1 Understanding

Game-like method increases the understanding of course content (Sillaots, 2015). The use of gamification is correlated with students’ learning outcomes (Veltsos, 2017). Students believe that gamification will help them to understand the course material better (Cheong et al., 2013). Games can enhance learning and understanding complex subjects (Surendeleg et al., 2014). Flipped approach is expected to improve students’ achievement of higher skills of Bloom’s taxonomy (Clark et al., 2016). Many students believe recorded lecture helping them to learn more effectively (Gilboy et al., 2015). Flipping the Excel course can improve students’ understanding and retention (Frydenberg, 2013).

3.1.2 Problem solving skills

Gamification forces students to think critically and to develop problem-solving skills (Cheong et al., 2013; Kanagasingam, 2017). Learning to write programs through games/contests develop students’ problem solving skills (Combefis et al., 2016). All students, exposed to gamification, agree or strongly agree that the class equip them with knowledge to solve real-world problems (Tan & Hew, 2016). Flipped approach encourages critical thinking (Tanner & Scott, 2015) and positively impact on students’ self-sufficiency (solving physics problems) (Aşıksoy & Özdamlı, 2016). It significantly heighten student behaviors such as problem solving (Clark et al., 2016).

3.1.3 Creativity

Creativity is one of subject matter interests when students choosing the MIS major (Rouibah, 2012). It is important to force students to think creatively. The utilization of gamification stimulates and inspires them to create new solutions (Kanagasingam, 2017). Students in the experimental group with gamification agree or strongly agree that they has many chances to exchange ideas (Tan & Hew, 2016). Flipped method lets students learn from each other and benefitting from perspectives such as creativity (Adnan, 2017).

3.1.4 Topic interests

Students in a blended-learning class with gamification agree or strongly agree that they are willing to study more about the content (Tan & Hew, 2016). Students in the gamified class like to attend the extra lectures (losup & Epema, 2014). Students in the gamified class enjoy playing games to learn materials (Poole et al., 2014). Students in gamification learning actively engage in contents (Surendeleg et al., 2014). Flipped instructions improve students’ knowledge on the topics (Adnan, 2017). Therefore, this work proposes the following hypotheses:

H1:
There is a significant difference between the ratings of participants’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) before and after participating in gamification.
H2:
There is a significant difference between the ratings of conductors’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) before and after leading gamification.
H3:
There is a significant difference between the ratings of participants’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) before and after participating in flipped classroom.
H4:
There is a significant difference between the ratings of conductors’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) before and after leading flipped classroom.
H5:
There is a significant difference between the ratings of participants’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) of the gamified group and the flipped group.
H6:
There is a significant difference between the ratings of conductors’ perceived usefulness (a) understanding, b) problem solving skills, c) creativity, d) topic interests) of the gamified group and the flipped group.

3.2 Engagement intention

Behavioral intention determines an individual’s behavior. In this study, students’ engagement intention are explored in three sub-aspects that are their intention to engage gamification or flipped classroom, intention to attend class, and intention to participate in the class.

3.2.1 Game/flipped classroom engagement intention

Gratifications of games could engage students (Hopkins et al., 2017). Students feel that the gamified class is engaging (Poole et al., 2014). A fun game increases students’ extra motivation (Aleksic-Maslac et al., 2017). Students exposed to gamified assignments agree or strongly agree that they are motivated to play games (Armier Jr et al., 2016). Flipped classroom engages students (Adnan, 2017; Clark et al., 2016; Oluwagbemi et al., 2014). It creates learner-centered environments (Tanner & Scott, 2015). Integrating clickers into instructions increase student engagement (Hung, 2017).

3.2.2 Class attendance intention

Gamification significantly improves student attendance (Caton & Greenhill, 2013; Fotaris et al., 2015; Kanagasingam, 2017), making classes more interesting (Cheong et al., 2013). Students in the gamified group look forward to attend this class (Tan & Hew, 2016). The flipped approach is expected to improve the classroom experience (Adnan, 2017) and reduce the passive class attendance (Clark et al., 2016).

3.2.3 Class participation intention

Gamification increases in-class participation (losup & Epema, 2014). It leads to greater participation in the discussion forum (Tan & Hew, 2016). Students will engage more when game mechanics are included (Armier Jr et al., 2016). Students in the gamified group show higher levels of participation (Poole et al., 2014). Flipped classroom enhances in-class engagement (Adnan, 2017). The flipped classroom with clickers gains more student participation (Hung, 2017). Thus, this work proposes the following hypotheses:

H7:
There is a significant difference between the ratings of participants’ game intention (a) engagement intention, b) class attendance intention, c) class participation intention) before and after participating in gamification.
H8:
There is a significant difference between the ratings of conductors’ game intention (a) engagement intention, b) class attendance intention, c) class participation intention) before and after leading gamification.
H9:
There is a significant difference between the ratings of participants’ flipped classroom intention (a) engagement intention, b) class attendance intention, c) class participation intention) intention before and after participating in flipped classroom.
H10:
There is a significant difference between the ratings of conductors’ flipped classroom intention (a) engagement intention, b) class attendance intention, c) class participation intention) before and after leading flipped classroom.
H11:
There is a significant difference between the ratings of participants’ game/flipped classroom intention (a) engagement intention, b) class attendance intention, c) class participation intention) of the gamified group and the flipped group.
H12:
There is a significant difference between the ratings of conductors’ game/flipped classroom intention (a) engagement intention, b) class attendance intention, c) class participation intention) of the gamified group and the flipped group.

3.3 Satisfaction

Students’ satisfaction significantly positively affects their continuance intention (Chow & Shi, 2014). Need satisfaction is significantly related to learners’ engagement (hours per week studying and number of hits), achievement (expected grade), and learning (perceived learning) in online environment (Chen & Jang, 2010). Satisfaction is also a significant driver of a virtual learning and education environment (VLE) continuance intention (Lin, 2012). Users’ satisfaction with e-learning positively impacts their intention to continually use e-learning (Lee, 2010).

Gamification is correlated with the increased student satisfaction (losup & Epema, 2014; Zhonggen & Guifang, 2016). Game elements are well accepted by students (Sillaots, 2015). Students enjoy the interactive nature of the flipped approach (Logan, 2015; Tanner & Scott, 2015). Students show satisfying learning experiences, received from the flipped approach (Adnan, 2017). Flipped classroom receives positive preferences from both faculty and learners (Tan et al., 2015). Flipped classroom creates high satisfaction with the course (Elliott, 2014; Frydenberg, 2013; Krueger & Storlie, 2015; Santiago Jr. et al., 2017). Students show positive attitudes in terms of satisfaction toward the inverted course (Dolgopolovas et al., 2014). Therefore, this work proposes the following hypotheses:

H13:
There is a significant difference between the ratings of participants’ game/flipped classroom satisfaction (a) enjoyment, b) content coverage, c) content reliability, d) conductors’ group unity, e) overall satisfaction) of the gamified group and the flipped group.

4. Research methodology

4.1 Participants and course design

This research is the quantitative research using pre-experimental study and survey research method to collect data. Participants in this study were convenient samples of undergraduate students enrolling in an introductory MIS course, taught by the same teacher (the researcher). Sample size was 46 students from two sections, which were randomly assigned student-centered learning methods: gamification or flipped approach. Two sections, using the same content/MIS textbooks, were opened on the same day in the same academic year. Gamification was applied to the first section, whereas the second section utilized the flipped approach. Backgrounds of students in both sections were quite the same. They were from various majors of the business school. None of them were familiar with gamification/flipped approaches before. They were informed about the research and the anonymity of data analysis at the first period and at the last period to summarize the benefits of gamification/flipped classroom.

Firstly, participants were instructed to divide voluntarily into 8 groups to be as conductors to lead gamified or flipped activities. Eight topics were randomly assigned to each group. They consisted of 1) Introduction/Information Systems in Organizations, 2) Hardware, 3) Software, 4) Telecommunications and Networks/The Internet, Intranets, and Extranets 5) Electronic and Mobile Commerce 6) Enterprise System/Information and Decision Support Systems/Knowledge Management System 7) Software Development, and 8) The Personal and social impacts. The class assignments were letting each group from both sections to gamify or flip the class activities in these topics for others who were not in the group subsequently.

Secondly, gamification conductors were opened to design and conduct free-format games after face-to-face lecturing by the teacher, whereas flipped classroom conductors were opened to design and conduct free-format presentations before face-to-face lecturing by the teacher. There were two Facebook groups: one for the gamified section and another one for the flipped section. For the flipped section, pre-class materials (sound clips and handouts) were added to the Facebook group in advance to help the conductor groups to prepare themselves. However, young students did not prefer the flipped classroom because they had low levels of preparation and expected to be taught instead (Clark et al., 2016). Therefore, the flipped section was designed to be partially flipped by the conductor groups only. Other students, who were not in the conductor groups of flipped classroom, were not strictly required to study materials beforehand. For the gamified section, nothing was given to the conductor groups in advance because they lead the gamification after face-to-face lecturing.

Thirdly, activities for each topic took around one hour and fifteen minutes. During the activities, articipants, who were not in the conductor groups, were freely to choose their levels of interaction (Birkenkrahe & Kjellin, 2015), to play/join the activities or only observe them and evaluate their peers at the end of each period. The activities had to satisfy participants under 5 criteria: enjoyment, content coverage, content reliability, conductors’ group unity, and overall satisfaction. Those criteria were specified in the course outline. A teacher turned her roles to correct misconceptions, give feedbacks, and answers questions (Clark et al., 2016; Tolks et al., 2016) for the conductor groups. Finally, after face-to-face teaching, both sections received the updated lecture clips, based on the same PowerPoint slides, through their Facebook groups.

4.2 Data collection and analysis

Data were collected via two questionnaires. The first questionnaire was applied to evaluate conductors from the view of participants and assess the pre-and post-perceptions of participants themselves. The first questionnaire was collected 8 times after each activity finish. The second questionnaire was used to examine the pre-and post-perceptions of conductors. The second questionnaire was collected only once after all activities finish. At the end of each activity, the first questionnaire was collected from participants. It contained the questions: a student’s ID, name, group ID, evaluation scores (from 0 to 5) in five criteria specified above, and his/her perceptions in terms of perceived usefulness and engagement intention before and after joining game/flipped classroom. Five question sets about perceived usefulness (understanding, problem solving skills, creativity, topic interests) and three question sets about engagement intention (game/flipped classroom engagement intention, class attendance intention, and class participation intention) were asked, using 6-point Likert scale. A question set referred to pre-and post-questions.

Retrospective pre-post test was applied to minimize the response shift bias occurring in the traditional pre-post design (Sánchez-Mendiola et al., 2015). At the end of activities, the second questionnaire was obtained from conductors once. The second questionnaire consisted of the questions: a student’s ID, name, gender, group ID, and his/her perceived usefulness and engagement intention (as same as in the first questionnaire but from the view of conductors). Perceived usefulness questions were developed from literature studies (Cheong et al., 2013; Kanagasingam, 2017; Kenny et al., 2017; Surendeleg et al., 2014; Tan & Hew, 2016). Engagement intention questions were adapted from the study of Ibanez, Di Serio, and Delgado Kloos (2014) and literature reviews. Satisfaction criteria (questions) were in accord with the course goals (Sillaots, 2015).

For data from participants (the first questionnaire), they would be analyzed for 8 times (8 topics). For data from conductors (the second questionnaire), they would be analyzed once because each student lead activities once. Firstly, data normality would be examined using Shapiro-Wilk test since the sample size was less than 50 (Ghasemi & Zahediasl, 2012). Then, pre- and post-test data of the gamified/flipped section would be compared separately using paired-sample $t$ -test. Lastly, data from the gamified section would be compared to data from the flipped section using independent sample $t$ -test. Nonparametric tests would be applied if the data were non-normal. In addition, since the sample size was quite small, non-parametric tests were generally suitable.

5. Results

There were 46 undergraduate students registered on the introductory MIS course. The gamified section had 26 students (15 females and 11 males), whereas the flipped section had 20 students (12 females and 8 males). Most of them were second-year students, which had not yet declared their major of study. They were just passed the general education courses and some introductory courses in the first year. Their approximate ages were between 19 and 21 years. Samples in both groups were similar in terms of age, gender, and academic level. Students in both sections also shared the similar levels of past academic performance. The nonparametric independent samples $t$ -test, the Mann-Whitney test, indicated that the cumulative GPA of students from the first section (Mean Rank $=$ 23.25) were equal to the cumulative GPA of students from the second section (Mean Rank $=$ 22.66), U $=$ 240.5, $p=$ 0.881. The normality test of data showed the non-normal distribution of data. Thus, nonparametric test (the Wilcoxon Signed-Rank Test and the Mann-Whitney U test) were applied.

5.1 Perceived usefulness

See Table 1.

5.1.1 Understanding

A Wilcoxon Signed-Ranks test for 7 of 8 gamified topics indicated that the post-understanding of participants was significantly rated more than their pre-understanding (1) Z $=$ $-$ 3.578, $p<$ 0.001; 2) Z $=$ $-$ 3.116, $p<$ 0.005; 3) Z $=$ $-$ 3.758, $p<$ 0.001; 4) Z $=$ $-$ 3.819, $p<$ 0.001; 5) Z $=$ $-$ 3.153, $p<$ 0.005; 6) Z $=$ $-$ 3.398, $p<$ 0.005; 7) Z $=$ $-$ 2.673, $p<$ 0.05; 8) Z $=$ $-$ 1.941, $p=$ 0.052). It also indicated that in the post-understanding of conductors was significantly rated more than their pre-understanding (Z $=$ $-$ 4.373, $p<$ 0.001). Hence, the hypothesis H1a was partially supported and the hypothesis H2a was supported. A Wilcoxon Signed-Ranks test for 6 of 8 flipped topics pointed out that the post-understanding

Table 1
Summary results in terms of perceived usefulness

Methods	Hypotheses	Understanding	Problem solving skills	Creativity	Topic interests
Gamification	The ratings of participants (before $\neq$ after)	Partially supported (after $>$ before)	Partially supported (after $>$ before)	Partially supported (after $>$ before) Weakly supported (before $>$ after)	Partially supported (after $>$ before)
	The ratings of conductors (before $\neq$ after)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)
Flipped classroom	The ratings of participants (before $\neq$ after)	Partially supported (after $>$ before)	Partially supported (after $>$ before)	Partially supported (after $>$ before)	Partially supported (after $>$ before)
	The ratings of conductors (before $\neq$ after)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)
Gamification vs. Flipped classroom	The ratings of participants (gamified $\neq$ flipped)	Not supported	Weakly supported (gamified $>$ flipped)	Not supported	Weakly supported (gamified $>$ flipped)
	The ratings of conductors (gamified $\neq$ flipped)	Not supported	Not supported	Not supported	Not supported

of participants was significantly rated more than their pre-understanding (1) Z $=$ $-$ 2.994, $p<$ 0.005; 2) Z $=$ $-$ 3.002, $p<$ 0.005; 3) Z $=$ $-$ 1.983, $p<$ 0.05; 4) Z $=$ $-$ 2.848, $p<$ 0.005; 5) Z $=$ $-$ 2.521, $p<$ 0.05; 6) Z $=$ $-$ 1.809, $p=$ 0.070; 7) Z $=$ $-$ 2.889, $p<$ 0.05; 8) Z $=$ $-$ 1.857, $p=$ 0.063). It also pointed out that the post-understanding of conductors was significantly rated more than their pre-understanding (Z $=$ $-$ 3.711, $p<$ 0.001). Thus, the hypothesis H3a was partially supported and the hypothesis H4a was supported. A Mann-Whitney U test for all topics indicated that there were no significant differences between participants’ understanding of the gamified section and the flipped section (1) U $=$ 110.0, $p=$ 0.682; 2) U $=$ 108, $p=$ 0.321; 3) U $=$ 95.5, $p=$ 0.679; 4) U $=$ 120.5, $p=$ 0.335; 5) U $=$ 86.5, $p=$ 0.726; 6) U $=$ 84.0, $p=$ 0.134; 7) U $=$ 81.0, $p=$ 0.122; 8) U $=$ 62.0, $p=$ 0.949). It also showed that there were no significant differences between conductors’ understanding of the gamified section and the flipped section (U $=$ 211.0, $p=$ 0.220). Therefore, the hypotheses H5a and H6a were not supported.

5.1.2 Problem solving skills

A Wilcoxon Signed-Ranks test for 6 of 8 gamified topics indicated that the post-problem solving skills of participants was significantly higher than their pre-problem

solving skills (1) Z $=$ $-$ 3.464, $p<$ 0.005; 2) Z $=$ $-$ 1.387, $p=$ 0.166; 3) Z $=$ $-$ 2.919, $p<$ 0.005; 4) Z $=$ $-$ 2.714, $p<$ 0.05; 5) Z $=$ $-$ 2.913, $p<$ 0.005; 6) Z $=$ $-$ 3.260, $p<$ 0.005; 7) Z $=$ $-$ 3.357, $p<$ 0.05; 8) Z $=$ $-$ 1.578, $p=$ 0.115). It also indicated that in the post-problem solving skills of conductors was significantly higher than their pre- problem solving skills (Z $=$ $-$ 4.456, $p<$ 0.001). Hence, the hypothesis H1b was partially supported and the hypothesis H2b was supported. A Wilcoxon Signed-Ranks test for 7 of 8 flipped topics pointed out that the post- problem solving skills of participants was significantly higher than their pre- problem solving skills (1) Z $=$ $-$ 2.414, $p<$ 0.05; 2) Z $=$ $-$ 2.970, $p<$ 0.005; 3) Z $=$ $-$ 2.121, $p<$ 0.05; 4) Z $=$ $-$ 2.636, $p<$ 0.05; 5) Z $=$ $-$ 2.041, $p<$ 0.05; 6) Z $=$ $-$ 2.165, $p<$ 0. 05; 7) Z $=$ $-$ 2.646, $p<$ 0.05; 8) Z $=$ $-$ 1.890, $p=$ 0.059). It also pointed out that the post- problem solving skills of conductors was significantly higher than their pre- problem solving skills (Z $=$ $-$ 3.456, $p<$ 0.005). Thus, the hypothesis H3b was partially supported and the hypothesis H4b was supported. A Mann-Whitney U test for 7 of 8 topics indicated that there were no significant differences between participants’ problem solving skills of the gamified section and the flipped section. A Mann-Whitney U test of one topic pointed out that the problem solving skills of participants in the gamified group was significantly higher than the problem solving skills of participants in the flipped group (1) U $=$ 100.0, $p=$ 0.426; 2) U $=$ 96.5, $p=$ 0.159; 3) U $=$ 95.0, $p=$ 0.672; 4) U $=$ 124.0, $p=$ 0.412; 5) U $=$ 85.0, $p=$ 0.667; 6) U $=$ 87.0, $p=$ 0.162; 7) U $=$ 63.5, $p<$ 0.05; 8) U $=$ 60.0, $p=$ 0.847). It also showed that there were no significant differences between conductors’ problem solving skills of the gamified section and the flipped section (U $=$ 179.5, $p=$ 0.062). Therefore, the hypothesis H5b was weakly supported and H6b were not supported.

5.1.3 Creativity

A Wilcoxon Signed-Ranks test for 7 of 8 gamified topics indicated that the post-creativity of participants was significantly rated more than their pre- creativity. But post-creativity of participants of one topic was significantly rated less than pre-creativity (1) Z $=$ $-$ 2.652, $p<$ 0.05; 2) Z $=$ $-$ 3.127, $p<$ 0.005; 3) Z $=$ $-$ 3.071, $p<$ 0.005; 4) Z $=$ $-$ 1.732, $p=$ 0.083; 5) Z $=$ $-$ 1.999, $p<$ 0.05; 6) Z $=$ $-$ 2.780, $p<$ 0.05; 7) Z $=$ $-$ 3.357, $p<$ 0.005; 8) Z $=$ $-$ 2.810, $p<$ 0.05). However, there was only one student who gave the negative perception. It also indicated that in the post-creativity of conductors was significantly rated more than their pre- creativity (Z $=$ $-$ 3.806, $p<$ 0.001). Hence, the hypothesis H1c was partially supported and the hypothesis H2c was supported. A Wilcoxon Signed-Ranks test for 7 of 8 flipped topics pointed out that the post-creativity of participants was significantly rated more than their pre-creativity (1) Z $=$ $-$ 2.588, $p<$ 0.05; 2) Z $=$ $-$ 2.392, $p<$ 0.05; 3) Z $=$ $-$ 2.449, $p<$ 0.05; 4) Z $=$ $-$ 2.496, $p<$ 0.05; 5) Z $=$ $-$ 2.060, $p<$ 0.05; 6) Z $=$ $-$ 2.326, $p<$ 0.05; 7) Z $=$ $-$ 2.887, $p<$ 0.005; 8) Z $=$ $-$ 1.857, $p=$ 0.063). It also pointed out that the post- creativity of conductors was significantly rated more than their pre- creativity (Z $=$ $-$ 3.460, $p<$ 0.005). Thus, the hypothesis H3c was partially supported and the hypothesis H4c was supported. A Mann-Whitney U test for all topics indicated that there were no significant differences between participants’ creativity of the gamified section and the flipped section (1) U $=$ 110.0, $p=$ 0.699; 2) U $=$ 98, $p=$ 0.164; 3) U $=$ 100.0, $p=$ 0.832; 4) U $=$ 128.0, $p=$ 0.502; 5) U $=$ 86.5, $p=$ 0.711; 6) U $=$ 84.0, $p=$ 0.127; 7) U $=$ 86.0, $p=$ 0.189; 8) U $=$ 45.5, $p=$ 0.268). It also showed that there were no significant differences between conductors’ creativity of the gamified section and the flipped section (U $=$ 239.5, $p=$ 0.691). Therefore, the hypotheses H5c and H6c were not supported.

5.1.4 Topic interests

A Wilcoxon Signed-Ranks test for 7 of 8 gamified topics indicated that the post-topic interests of participants was significantly rated more than their pre-topic interests (1) Z $=$ $-$ 2.588, $p<$ 0.05; 2) Z $=$ $-$ 2.352, $p<$ 0.05; 3) Z $=$ $-$ 2.360, $p<$ 0.05; 4) Z $=$ $-$ 1.795, $p=$ 0.073; 5) Z $=$ $-$ 2.889, $p<$ 0.005; 6) Z $=$ $-$ 2.968, $p<$ 0.005; 7) Z $=$ $-$ 2.714, $p<$ 0.05; 8) Z $=$ $-$ 2.840, $p<$ 0.05). It also indicated that in the post-topic interests of conductors was significantly rated more than their pre-topic interests (Z $=$ $-$ 4.142, $p<$ 0.001). Hence, the hypothesis H1d was partially supported and the hypothesis H2d was supported. A Wilcoxon Signed-Ranks test for 7 of 8 flipped topics pointed out that the post-topic interests of participants was significantly rated more than their pre-topic interests (1) Z $=$ $-$ 2.694, $p<$ 0.05; 2) Z $=$ $-$ 2.395, $p<$ 0.05; 3) Z $=$ $-$ 2.919, $p<$ 0.005; 4) Z $=$ $-$ 2.840, $p<$ 0.05; 5) Z $=$ $-$ 2.646, $p<$ 0.05; 6) Z $=$ $-$ 2.326, $p<$ 0. 05; 7) Z $=$ $-$ 3.051, $p<$ 0.005; 8) Z $=$ $-$ 1.841, $p=$ 0.066). It also pointed out that the post-topic interests of conductors was significantly rated more than their pre-topic interests (Z $=$ $-$ 3.684, $p<$ 0.001). Thus, the hypothesis H3d was partially supported and the hypothesis H4d was supported. A Mann-Whitney U test for 7 of 8 topics indicated that there were no significant differences between participants’ topic interests of the gamified section and the flipped section. A Mann-Whitney U test of one topic pointed out that the topic interests of participants in the gamified group was significantly higher than the topic interests of participants in the flipped group (1) U $=$ 75.5, $p=$ 0.064; 2) U $=$ 87.5, $p=$ .072; 3) U $=$ 91.5, $p=$ 0.541; 4) U $=$ 111.0, $p=$ 0.197; 5) U $=$ 87.0, $p=$ 0.737; 6) U $=$ 88.5, $p=$ 0.178; 7) U $=$ 46.5, $p<$ 0.005; 8) U $=$ 52.0, $p=$ 0.487). It also showed that there were no significant differences between conductors’ topic interests of the gamified section and the flipped section (U $=$ 251.0, $p=$ 0.889). Therefore, the hypothesis H5d was weakly supported and H6d were not supported.

5.2 Engagement intention

See Table 2.

5.2.1 Game/flipped classroom engagement intention

A Wilcoxon Signed-Ranks test for 5 of 8 gamified topics indicated that the post-game engagement intention of participants was significantly rated more than their pre-game engagement intention (1) Z $=$ $-$ 3.442, $p<$ 0.005; 2) Z $=$ $-$ 1.513, $p=$ 0.130; 3) Z $=$ $-$ 3.140, $p<$ 0.005; 4) Z $=$ $-$ 0.144, $p=$ 0.885; 5) Z $=$ $-$ 2.178, $p<$ 0.05; 6) Z $=$ $-$ 2.642, $p<$ 0. 05; 7) Z $=$ $-$ 2.271, $p<$ 0.05; 8) Z $=$ $-$ 2.309, $p<$ 0.05). But post-game engagement intention of participants of one topic was also significantly rated less than pre-game engagement intention. Nevertheless, there was only one student who gave the negative perception. It also indicated that in the post-game engagement intention of conductors was significantly rated more than their pre-game engagement intention (Z $=$ $-$ 3.673, $p<$ 0.001). Hence, the hypothesis H7a was partially supported and the hypothesis H8a was supported. A Wilcoxon Signed-Ranks test for 3 of 8 flipped topics pointed out that the post-flipped classroom engagement intention of participants was significantly rated more than their pre-flipped classroom engagement intention (1) Z $=$ $-$ 2.489, $p<$ 0.05; 2) Z $=$ $-$ 2.506, $p<$ 0.05; 3) Z $=$ $-$ 2.714, $p<$ 0.05; 4) Z $=$ $-$ 1.931, $p=$ 0.053; 5) Z $=$ $-$ 1.730, $p=$ 0.084; 6) Z $=$ $-$ 1.508, $p=$ 0.132; 7) Z $=$ $-$ 1.350, $p=$ 0.177; 8) Z $=$ $-$ 1.414, $p=$ 0.157). It also pointed out that the post-flipped classroom engagement intention of conductors was significantly rated more than their pre-flipped classroom engagement intention (Z $=$ $-$ 3.250, $p<$ 0.005). Thus, the hypothesis H9a was weakly supported and the hypothesis H10a was supported. A Mann-Whitney U test for 6 of 8 topics indicated that there were no significant differences between participants’ game/flipped classroom engagement intention of the gamified section and the flipped section. A Mann-Whitney U test of two topics indicated that the game/flipped classroom engagement intention of participants in the gamified group was significantly higher than the game/flipped classroom engagement intention of participants in the flipped group (1) U $=$ 107.5, $p=$ 0.621; 2) U $=$ 116.0, $p=$ 0.501; 3) U $=$ 68.5, $p=$ 0.101; 4) U $=$ 137.0, $p=$ 0.720; 5) U $=$ 73.5, $p=$ 0.307; 6) U $=$ 39.5, $p<$ 0.005; 7) U $=$

Table 2
Summary results in terms of engagement intention

Methods	Hypotheses	Engagement intention	Class attendance intention	Class participation intention
Gamification	The ratings of participants (before $\neq$ after)	Partially supported (after $>$ before) Weakly supported (before $>$ after)	Weakly supported (after $>$ before)	Partially supported (after $>$ before)
	The ratings of conductors (before $\neq$ after)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)
Flipped classroom	The ratings of participants (before $\neq$ after)	Weakly supported (after $>$ before)	Weakly supported (after $>$ before)	Partially supported (after $>$ before)
	The ratings of conductors (before $\neq$ after)	Supported (after $>$ before)	Supported (after $>$ before)	Supported (after $>$ before)
Gamification vs. Flipped classroom	The ratings of participants (gamified $\neq$ flipped)	Weakly supported (gamified $>$ flipped)	Weakly supported (gamified $>$ flipped)	Weakly supported (gamified $>$ flipped)
	The ratings of conductors (gamified $\neq$ flipped)	Not supported	Not supported	Not supported

53.5, $p<$ 0.05; 8) U $=$ 41.5, $p=$ 0.173). It also showed that there were no significant differences between conductors’ game/flipped classroom engagement intention of the gamified section and the flipped section (U $=$ 174.0, $p=$ 0.051). Therefore, the hypothesis H11a was weakly supported and the hypothesis H12a were not supported.

5.2.2 Class attendance intention

A Wilcoxon Signed-Ranks test for 3 of 8 gamified topics indicated that the post-class attendance intention of participants was significantly rated more than their pre- class attendance intention (1) Z $=$ $-$ 1.998, $p<$ 0.05; 2) Z $=$ $-$ 1.000, $p=$ 0.317; 3) Z $=$ $-$ 2.799, $p<$ 0.05; 4) Z $=$ $-$ 0.486, $p=$ 0.627; 5) Z $=$ $-$ 1.933, $p=$ 0.053; 6) Z $=$ $-$ 2.495, $p<$ 0.05; 7) Z $=$ $-$ 2.121, $p<$ 0.05; 8) Z $=$ $-$ 1.809, $p=$ 0.070). It also indicated that in the post-class attendance intention of conductors was significantly rated more than their pre-class attendance intention (Z $=$ $-$ 3.595, $p<$ 0.001). Hence, the hypothesis H7b was weakly supported and the hypothesis H8b was supported. A Wilcoxon Signed-Ranks test for 3 of 8 flipped topics pointed out that the post-class attendance intention of participants was significantly rated more than their pre-class attendance intention (1) Z $=$ $-$ 2.558, $p<$ 0.05; 2) Z $=$ $-$ 1.812, $p=$ .070; 3) Z $=$ $-$ 1.857, $p=$ 0.063; 4) Z $=$ $-$ 2.887, $p<$ 0.005; 5) Z $=$ $-$ 1.930, $p=$ 0.054; 6) Z $=$ $-$ 2.111, $p<$ .05; 7) Z $=$ $-$ 0.541, $p=$ 0.589; 8) Z $=$ $-$ 1.841, $p=$ 0.066). It also pointed out that the post-class attendance intention of conductors was significantly rated more than their pre-class attendance intention (Z $=$ $-$ 3.270, $p<$ 0.005). Thus, the hypothesis H9b was weakly supported and the hypothesis H10b was supported. A Mann-Whitney U test for 6 of 8 topics indicated that there were no significant differences between participants’ class attendance intention of the gamified section and the flipped section. A Mann-Whitney U test of three topics indicated that the class attendance intention of participants in the gamified group was significantly higher than the class attendance intention of participants in the flipped group (1) U $=$ 103.0, $p=$ 0.501; 2) U $=$ 118.0, $p=$ 0.559; 3) U $=$ 61.5, $p<$ 0.05; 4) U $=$ 112.0, $p=$ 0.214; 5) U $=$ 87.0, $p=$ 0.746; 6) U $=$ 62.0, $p<$ 0.05; 7) U $=$ 38.0, $p<$ 0.005; 8) U $=$ 53.0, $p=$ 0.517). It also showed that there were no significant differences between conductors’ class attendance intention of the gamified section and the flipped section (U $=$ 197.5, $p=$ 0.159). Therefore, the hypothesis H11b was weakly supported and the hypothesis H12b was not supported.

5.2.3 Class participation intention

A Wilcoxon Signed-Ranks test for 5 of 8 gamified topics indicated that the post-class participation intention of participants was significantly rated more than their pre-class participation intention (1) Z $=$ $-$ 3.626, $p<$ 0.001; 2) Z $=$ $-$ 2.111, $p<$ 0.05; 3) Z $=$ $-$ 2.543, $p<$ 0.05; 4) Z $=$ $-$ 0.575, $p=$ 0.565; 5) Z $=$ $-$ 1.933, $p=$ 0.053; 6) Z $=$ $-$ 3.051, $p<$ 0.001; 7) Z $=$ $-$ 2.309, $p<$ 0.05; 8) Z $=$ $-$ 2.178, $p<$ 0.05). It also indicated that in the post-class participation intention of conductors was significantly rated more than their pre-class participation intention (Z $=$ $-$ 3.700, $p<$ 0.001). Hence, the hypothesis H7c was partially supported and the hypothesis H8c was supported. A Wilcoxon Signed-Ranks test for 5 of 8 flipped topics pointed out that the post- class participation intention of participants was significantly rated more than their pre- class participation intention (1) Z $=$ $-$ 2.676, $p<$ 0.05; 2) Z $=$ $-$ 2.354, $p<$ 0.05; 3) Z $=$ $-$ 2.333, $p<$ 0.05; 4) Z $=$ $-$ 2.496, $p<$ 0.05; 5) Z $=$ $-$ 1.561, $p=$ 0.119; 6) Z $=$ $-$ 0.954, $p=$ 0.340; 7) Z $=$ $-$ 2.066, $p<$ 0.05; 8) Z $=$ $-$ 1.890, $p=$ 0.059). It also pointed out that the post- class participation intention of conductors was significantly rated more than their pre- class participation intention (Z $=$ $-$ 3.167, $p<$ 0.005). Thus, the hypothesis H9c was partially supported and the hypothesis H10c was supported. A Mann-Whitney U test for 6 of 8 topics indicated that there were no significant differences between participants’ class participation intention of the gamified section and the flipped section. A Mann-Whitney U test of two topics pointed out that the class participation intention of participants in the gamified group was significantly higher than the class participation intention of participants in the flipped group (1) U $=$ 95.0, $p=$ 0.297; 2) U $=$ 99.5, $p=$ 0.200; 3) U $=$ 97.5, $p=$ 0.737; 4) U $=$ 95.0, $p=$ 0.055; 5) U $=$ 78.5, $p=$ 0.451; 6) U $=$ 50.0, $p<$ 0.005; 7) U $=$ 60.0, $p<$ 0.05; 8) U $=$ 57.0, $p=$ 0.699). It also showed that there were no significant differences between conductors’ class participation intention of the gamified section and the flipped section (U $=$ 251.0, $p=$ 0.896). Therefore, the hypothesis H11c was weakly supported and the hypothesis H12c was not supported.

5.3 Satisfaction

See Table 3.

5.3.1 Game/flipped classroom enjoyment

A Mann-Whitney U test for 6 of 8 topics indicated that there were no significant differences between participants’ game/flipped classroom enjoyment of the gamified section and the flipped section. A Mann-Whitney U test of two topics pointed out that the game enjoyment of participants in the gamified group was significantly higher than the flipped classroom enjoyment of participants in the flipped group (1) U $=$ 77.0, $p=$ 0.065; 2) U $=$ 102.5, $p=$ 0.236; 3) U $=$ 33.5, $p<$ 0.005; 4) U $=$ 110.5, $p=$ 0.183; 5) U $=$ 49.5, $p<$ 0.05; 6) U $=$ 83.5, $p=$ 0.121; 7) U $=$ 78.5, $p=$ 0.102; 8) U $=$ 60.0, $p=$ 0.845). Therefore, the hypothesis H13a was weakly supported.

5.3.2 Content coverage

A Mann-Whitney U test for 5 of 8 topics indicated that there were no significant differences between participants’ satisfaction in terms of content coverage of the gamified section and the flipped section. A Mann-Whitney U test of two topics pointed out that the satisfaction in terms of content coverage of participants

Table 3
Summary results in terms of satisfaction

Methods	Hypotheses	Enjoyment	Content coverage	Content reliability	Conductor’s group unity	Overall satisfaction
Gamification vs. Flipped classroom	The ratings of participants (gamified $\neq$ flipped)	Weakly supported (gamified $>$ flipped)	Weakly supported (both gamified $>$ flipped and flipped $>$ gamified)	Partially supported (both gamified $>$ flipped and flipped $>$ gamified)	Weakly supported (both gamified $>$ flipped and flipped $>$ gamified)	Partially supported (both gamified $>$ flipped and flipped $>$ gamified)

in the flipped group was significantly higher than the satisfaction in terms of content coverage of participants in the gamified group, but a Mann-Whitney U test of another topic indicated that the satisfaction in terms of content coverage of participants in the gamified group was significantly higher than the satisfaction in terms of content coverage of participants in the flipped group (1) U $=$ 39.5, $p<$ 0.001; 2) U $=$ 90.0, $p=$ 0.094; 3) U $=$ 98.0, $p=$ 0.736; 4) U $=$ 52.5, $p<$ 0.001; 5) U $=$ 90.0, $p=$ 0.859; 6) U $=$ 50.0, $p<$ 0.005; 7) U $=$ 117.0, $p=$ 1.000; 8) U $=$ 53.5, $p=$ 0.535). Therefore, the hypothesis H13b was weakly supported.

5.3.3 Content reliability

A Mann-Whitney U test for 3 of 8 topics indicated that there were no significant differences between participants’ satisfaction in terms of content reliability of the gamified section and the flipped section. A Mann-Whitney U test of three topics pointed out that the satisfaction in terms of content reliability of participants in the flipped group was significantly higher than the satisfaction in terms of content reliability of participants in the gamified group, but a Mann-Whitney U test of another two topics indicated that the satisfaction in terms of content reliability of participants in the gamified group was significantly higher than the satisfaction in terms of content reliability of participants in the flipped group (1) U $=$ 51.0, $p<$ 0.005; 2) U $=$ 75.5, $p<$ 0.05; 3) U $=$ 84.5, $p=$ 0.343; 4) U $=$ 80.0, $p<$ 0.05; 5) U $=$ 85.5, $p=$ 0.670; 6) U $=$ 71.0, $p<$ 0.05; 7) U $=$ 57.0, $p<$ 0.05; 8) U $=$ 44.5, $p=$ 0.230). Therefore, the hypothesis H13c was partially supported.

5.3.4 Conductor groups’ unity

A Mann-Whitney U test for 5 of 8 topics indicated that there were no significant differences between participants’ satisfaction in terms of conductor groups’ unity of the gamified section and the flipped section. A Mann-Whitney U test of a topic pointed out that the satisfaction in terms of conductor groups’ unity of participants in the flipped group was significantly higher than the satisfaction in terms of conductor groups’ unity of participants in the gamified group, but a Mann-Whitney U test of two topics indicated that the satisfaction in terms of conductor groups’ unity of participants in the gamified group was significantly higher than the satisfaction in terms of conductor groups’ unity of participants in the flipped group (1) U $=$ 44.0, $p<$ 0.005; 2) U $=$ 109.5, $p=$ 0.366; 3) U $=$ 50.5, $p<$ 0.05; 4) U $=$ 135.0, $p=$ 0.664; 5) U $=$ 73.5, $p=$ 0.315; 6) U $=$ 72.5, $p<$ 0.05; 7) U $=$ 74.0, $p=$ 0.064; 8) U $=$ 40.5, $p=$ 0.149). Therefore, the hypothesis H13e was weakly supported.

5.3.5 Overall satisfaction

A Mann-Whitney U test for 4 of 8 topics indicated that there were no significant differences between participants’ overall satisfaction of the gamified section and the flipped section. A Mann-Whitney U test of two topics pointed out that the overall satisfaction of participants in the flipped group was significantly higher than the overall satisfaction of participants in the gamified group, but a Mann-Whitney U test of another two topics indicated that the overall satisfaction of participants in the gamified group was significantly higher than the overall satisfaction of participants in the flipped group (1) U $=$ 118.0, $p=$ 0.964; 2) U $=$ 80.0, $p<$ 0.05; 3) U $=$ 100.5, $p=$ 0.841; 4) U $=$ 90.0, $p<$ 0.05; 5) U $=$ 75.0, $p=$ 0.351; 6) U $=$ 48.0, $p<$ 0.005; 7) U $=$ 62.5, $p<$ 0.05; 8) U $=$ 47.5, $p=$ 0.295). Therefore, the hypothesis H13e was partially supported.

In addition to student perceptions, grades of students in both sections were compared. The Mann-Whitney test, indicated that the grades of students from the gamified section (Mean Rank $=$ 22.54) were equal to the grades of students from the flipped section (Mean Rank $=$ 23.63), U $=$ 235.0, $p=$ 0.771.

6. Discussion

6.1 RQ1: Do student-led gamification positively impact perceived usefulness and engagement intention of game participants?

Conforming to the literature supports in the research hypotheses section, students’ perceived usefulness after participating student-led gamification is higher than their perceived usefulness before participating the gamified activities. Students’ engagement intention after engaging in student-led gamification in almost all cases is higher than their engagement intention before engaging in the gamified activities.

6.2 RQ2: Do student-led gamification promote perceived usefulness and engagement intention of game conductors?

Conforming to the literature supports in the research hypotheses section, game conductors’ perceived usefulness after leading gamified activities is higher than their perceived usefulness before leading the activities. In addition, students’ engagement intention after leading gamification is higher than their engagement intention before leading the gamified activities.

6.3 RQ3: Do student-led flipped classroom positively impact perceived usefulness and engagement intention of flipped classroom participants?

Conforming to the literature supports in the research hypotheses section, students’ perceived usefulness, after joining the student-led flipped activities, is higher than perceived usefulness before joining the activities. Participants’ engagement intention after engaging in student-led flipped classroom in every significant case is higher than their engagement intention before engaging in the flipped activities.

6.4 RQ4: Do student-led flipped classroom promote perceived usefulness and engagement intention of flipped classroom conductors?

Conforming to the literature supports in the research hypotheses section, flipped classroom conductors’ perceived usefulness after leading flipped activities is higher than their perceived usefulness before leading the activities. Conductors’ engagement intention after leading inverted classroom is higher than their engagement intention before leading the inverted activities.

6.5 RQ5: Are there any differences between perceived usefulness, engagement intention, and satisfaction of game/flipped classroom participants?

Conforming to the literature supports in the research hypotheses section, there are significant differences between the perceived usefulness in terms of problem solving skills and topic interests of game/flipped classroom participants, but there are no significant differences between their understanding and creativity. Participants’ problem solving skills and topic interests of gamified section is higher than the flipped section in the same topic. This is also supported by the evaluation scores of gamified section in the 7th topic, which are higher than the evaluation scores of the flipped section. Moreover, there is a research specifies that one drawback of the flipped classroom is the need of additional in-class time for problem solving (Clark et al., 2016).

There are significant differences between the engagement intention of game and flipped classroom participants. Gamified participants’ engagement intention is higher than flipped participants’ engagement intention in the two topics. This is confirmed by the evaluation scores of the gamified section in the 6th and the 7th topics, which are higher than the evaluation scores of the flipped section. Moreover, gamified participants’ enjoyment is higher than flipped participants’ enjoyment in the two same topics. This is also supported by the enjoyment scores of gamified section in the 3rd and the 5th topics, which are explicitly greater than the evaluation scores of flipped section. In terms of content coverage, content reliability, unity of conductor groups, and overall satisfaction, there are some cases that gamified section significantly receives higher rating and some cases that flipped section significantly gains higher scores. Total satisfaction scores clearly supports the result. This can also be explained by some students that do not prefer flipped classroom over traditional classroom (Clark et al., 2016; Mikkelsen, 2015). In addition, there are many types of students in the classroom. Gamified activities may be more appealing to super-achiever or performance-oriented people than non-achiever people (Tan & Hew, 2016).

6.6 RQ6: Are there any differences between perceived usefulness and engagement intention of game/ flipped classroom conductors?

There are no statistical differences between the perceived usefulness and engagement intention of game/ flipped classroom conductors. The results are supported by the literature study indicating that the flipped approach may not increase learning engagement and satisfaction of students (Adnan, 2017).

Comparing between two approaches, gamification receives participants’ perceptions of understanding and creativity as equal to the flipped approach. In terms of problem solving skills, topic interests, game/flipped classroom engagement intention, class attendance intention, and class participation intention, there are some evidences confirming that gamification yields better results than flipped classroom. Gamified classroom also receives higher satisfaction than the flipped classroom in terms of enjoyment. But for other aspects, which are content coverage, content reliability, unity of conductor groups, and overall satisfaction, the satisfactions of participants in these factors between two approaches are higher in some topics and lower in some topics, in line with the overall satisfaction score of each topic. This study gives the contrast result from the study of Sillaots (2015) specifying that students do not feel the need for gamified learning activities but they think that the flipped classroom approach can be useful. Regarding the perspectives of game/flipped classroom conductors, the gamified approach gains the same level of perceived usefulness and engagement intention as the flipped approach. Two teaching approaches can be applied together (Birkenkrahe & Kjellin, 2015; Yildirim, 2017). However, those combinations work only in some cases. Therefore, the findings from this research could be guidelines for selecting the right approach (if necessary), customizing the activities, or placing importance on one approach over another.

7. Conclusion and practical/theoretical contributions

This study explore the effectiveness of gamification and the flipped classroom in an introductory MIS course and provide early evidence about the comparison of perceived usefulness, engagement intention, and satisfaction between two methods from the viewpoints of participants and conductors. Student-led activities are assigned as team-based assignments. This work hypothesizes that both student-led approaches would lead to the higher perceived usefulness (TAM factor) and greater engagement intention (TAM/ECM factor). The researcher also expects that there are some differences regarding usefulness, engagement intention, and satisfaction (ECM factor) between the approaches. Findings support, partially support, or weakly support the first expectation. For the second expectation, gamification approach performs slightly better than the flipped approach in the participants’ opinions relating to the increase of problem solving skills, topic interests, game/flipped classroom engagement intention, class attendance intention, class participation intention, and satisfaction (game/flipped classroom enjoyment). The results could suggest educators in selecting and integrating student-centered learning methods into the introductory courses.

For theoretical contributions, this research applies two established theories in the IS field: TAM and ECM, to explore the effectiveness of two student-centered learning methods, which generally utilize IS/IT as facilitating tools. In addition, although some researchers have studied the gamification and the flipped classroom, only few of them have explored the perceived usefulness, engagement intention, and satisfaction of these approaches in various aspects. This study is the first study examining those factors, both from the viewpoints of participants and conductors. In addition, perceived usefulness, engagement intention, and satisfaction of two approaches are compared for the first time. Other studies can replicate this study for better selecting the student-centered teaching approaches for learners in the future.

For practical implications, since new pedagogical approaches are not easy to implement. In addition, teachers who feel unfamiliar with gamification worry about adopting games to their classes. Thus, this study guides teachers to apply student-led gamification as the project assignments. From students’ viewpoints, both participants and conductors, student-led gamification improves their understanding, problem solving, creativity, topic interests, game engagement intention, class attendance intention, and class participation intention. In terms of the flipped classroom, flipped classroom could be applied in the classroom as peer-to-peer teaching. It significantly increase students’ perceived usefulness and engagement intention both for participants and conductors. Using the student-led activities is not only relieving teachers’ burdens, but also creates fun and more suitable activities by their peers. In addition, the flipped classroom setting should be customized, listening to students (Birkenkrahe & Kjellin, 2015). Student-led activities may be better for varying in-class activities and understanding students more. However, these approaches may not be well-suited for all topics (Clark et al., 2016). For example, the topic ‘the personal and social impacts of computers’ may not be suitable to be flipped. Some topics such as ‘introduction/information systems in organizations’ may be suited for flipping or gamifying them. Instructors should select more simple topics to implement the student-led activities.

8. Limitations and future works

The generalization of findings should be done with caution due to the small sample sizes and using self-reported surveys. However, the experiments are repeated 8 times and data are collected from two viewpoints, so the present study yields some interesting results as described above. In the future, the study should be expanded to other introductory courses that have more students and to compare the effectiveness of applying gamification/flipped classroom to small and large classes. Qualitative study should be conducted to extend findings. The comparison between new pedagogical approaches and the traditional lecture should be conducted. Additional studies should examine more factors such as high order thinking skills. Other cutting-edge pedagogical techniques should be explored and compared. Negative effects of these new techniques should also be investigated.

References

Adnan,

(2017). Perceptions of senior-year ELT students for flipped classroom: a materials development course. Computer Assisted Language Learning, 30(3-4), 204-222. aaa(000)Alabbasi, D. (2017). Exploring Graduate Students’ Perspectives towards using Gamification Techniques in Online Learning. Turkish Online Journal of Distance Education-TOJDE, 18(3), 180-196. aaa(000)Aleksic-Maslac, K., Sinkovic, B., & Vranesic, P. (2017). Influence of gamification on student engagement in education. International Journal of Education and Learning Systems,, 2, 76-82. aaa(000)Armier Jr, D. D., Shepherd, C. E., & Skrabut, S. (2016). Using game elements to increase student engagement in course assignments. College Teaching, 64(2), 64-72.

Aşıksoy,

, & Özdamlı,

(2016). Flipped Classroom adapted to the ARCS Model of Motivation and applied to a Physics Course. Eurasia Journal of Mathematics, Science & Technology Education, 12(6), 1589-1603.

Birkenkrahe,

, & Kjellin,

(2015). Improving Student Interaction and Engagement in the Flipped Classroom. Paper presented at the 14th European Conference on e-Learning (ECEl2015), Hatfield, UK.

Buhl,

H. U.

, & Lehnert,

(2012). Information Systems and Business & Information Systems Engineering: Status Quo and Outlook. Paper presented at the International Conference on Business Information Systems (BIS), Vilnius, Lithuania.

Caton,

, & Greenhill,

(2013). The effects of gamification on student attendance and team performance in a third-year undergraduate game production module. Paper presented at the European conference on games based learning, Porto, Portugal.

Chen,

K.-C.

, & Jang,

S.-J.

(2010). Motivation in online learning: Testing a model of self-determination theory. Computers in Human Behavior, 26(4), 741-752.

Cheong,

Filippou,

, & Cheong,

(2013). Understanding Student Perceptions of Game Elements to Develop Gamified Systems for Learning. Paper presented at the Pacific Asia Conference on Information Systems (PACIS), Jeju Island, Korea.

Chow,

W. S.

, & Shi,

(2014). Investigating students’ satisfaction and continuance intention toward e-learning: An Extension of the expectation – confirmation model. Procedia-Social and Behavioral Sciences, 141, 1145-1149.

Clark,

R. M.

Besterfield-Sacre,

Budny,

Bursic,

K. M.

Clark,

W. W.

Norman,

B. A.

Patzer,

J. F.

(2016). Flipping Engineering Courses: A School Wide Initiative. Advances in Engineering Education, 5(3), 1-39.

10.

Combefis,

Bersnevicius,

, & Dagiene,

(2016). Learning Programming through Games and Contests: Overview, Characterisation and Discussion. Olympiads in Informatics, 10, 39-60.

11.

Davis,

F. D.

(1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. Mis Quarterly, 13(3), 319-340.

12.

Dolgopolovas,

Savulionienė,

, & Dagienė,

(2014). Enhancing students’ motivation in the inverted CS2 course: A Case study. Paper presented at the The International Conference on e-Learning 2014, Tenerife, Spain.

13.

Elliott,

(2014). Do students like the flipped classroom? An investigation of student reaction to a flipped undergraduate IT course. Paper presented at the 2014 IEEE Frontiers in Education Conference (FIE) Proceedings.

14.

Estévez-Ayres,

Alario-Hoyos,

Pérez-Sanagustín,

Pardo,

. Crespo-García,

R. M.

Leony,

, & Delgado-Kloos,

(2015). A methodology for improving active learning engineering courses with a large number of students and teachers through feedback gathering and iterative refinement. International Journal of Technology and Design Education, 25(3), 387-408.

15.

Ferreira,

(2014). The Maritime English MOOC: Using MOOC Technology to flip the Classroom. Paper presented at the International Maritime English Conference (IMEC 2014), Maritime Institute Willem Barentsz, Terchelling, the Netherlands.

16.

Fotaris,

Mastoras,

Leinfellner,

, & Rosunally,

(2015). Who wants to be a Pythonista? Using Gamification to Teach Computer Programming. Paper presented at the 7th International Conference on Education and New Learning Technologies, Barcelona, Spain.

17.

Frania,

(2014). New educational Trends Connected with the development of Media and Innovative Technologies – A Few reflections on the Future Perspectives on Learning and Teaching. Journal of Educational and Social Research, 4(4), 232-236.

18.

Frydenberg,

(2013). Flipping excel. Information Systems Education Journal, 11(1), 63-73.

19.

Ghasemi,

, & Zahediasl,

(2012). Normality tests for statistical analysis: a guide for non-statisticians. International Journal of Endocrinology and Metabolism, 10(2), 486-489.

20.

Gilboy,

M. B.

Heinerichs,

, & Pazzaglia,

(2015). Enhancing student engagement using the flipped classroom. Journal of Nutrition Education and Behavior, 47(1), 109-114.

21.

Hamari,

, & Koivisto,

(2015). Why do people use gamification services? International Journal of Information Management, 35(4), 419-431.

22.

Hamari,

Shernoff,

D. J.

Rowe,

Coller,

Asbell-Clarke,

, & Edwards,

(2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170-179.

23.

Hong,

S.-J.

Thong,

, & Tam,

K.-Y.

(2005). Understanding continued IT usage: An extension to the expectation-confirmation model in IT domain. Paper presented at the Pacific Asia Conference on Information Systems (PACIS 2005), Bangkok, Thailand.

24.

Hopkins,

, et al., (2017). To the point: medical education, technology, and the millennial learner. American Journal of Obstetrics and Gynecology, 218(2), 188-192.

25.

Hung,

H.-T.

(2017). Clickers in the flipped classroom: bring your own device (BYOD) to promote student learning. Interactive Learning Environments, 25(8), 983-995.

26.

Ibanez,

Di Serio,

, & Delgado Kloos,

(2014). Gamification for Engaging Computer Science Students in Learning Activities: A Case Study. IEEE Transactions on Learning Technologies, 7(3), 291-301.

27.

Joo,

Y. J.

Lim,

K. Y.

, & Kim,

E. K.

(2011). Online university students’ satisfaction and persistence: Examining perceived level of presence, usefulness and ease of use as predictors in a structural model. Computers & Education, 57(2), 1654-1664.

28.

Kanagasingam,

(2017). Increasing the Learning Motivations Using Gamification. Journal of Administrative Management, Education and Training, 13(1), 192-198.

29.

Kenny,

Lyons,

, & Lynn,

(2017). Don’t Make the Player, Make the Game: Exploring the Potential of Gamification in IS Education. Paper presented at the Americas Conference on Information Systems (AMCIS), Boston, MA. https://aisel.aisnet.org/.

30.

Krueger,

G. B.

, & Storlie,

C. H.

(2015). Evaluation of a Flipped Classroom Format for an Introductory-Level Marketing Class. Journal of Higher Education Theory and Practice, 15(1), 19-36.

31.

Kuo,

M.-S.

, & Chuang,

T.-Y.

(2016). How gamification motivates visits and engagement for online academic dissemination – An empirical study. Computers in human Behavior, 55(Part A), 16-27.

32.

Lameras,

, & Moumoutzis,

(2015). Towards the gamification of inquiry-based flipped teaching of mathematics a conceptual analysis and framework. Paper presented at the International Conference on Interactive Mobile Communication Technologies and Learning (IMCL 2015), Thessaloniki, Greece.

33.

Latulipe,

Long,

N. B.

, & Seminario,

C. E.

(2015). Structuring flipped classes with lightweight teams and gamification. Paper presented at the the 46th ACM Technical Symposium on Computer Science Education (SIGCSE), Kansas City, Missouri, USA.

34.

Lee,

M.-C.

(2010). Explaining and predicting users’ continuance intention toward e-learning: An extension of the expectation – confirmation model. Computers & Education, 54(2), 506-516.

35.

Lin,

W.-S.

(2012). Perceived fit and satisfaction on web learning performance: IS continuance intention and task-technology fit perspectives. International Journal of Human-Computer Studies, 70(7), 498-507.

36.

Liu,

I.-F.

Chen,

M. C.

Sun,

Y. S.

Wible,

, & Kuo,

C.-H.

(2010). Extending the TAM model to explore the factors that affect Intention to Use an Online Learning Community. Computers & Education, 54(2), 600-610.

37.

Llorens-Largo,

Villagrá-Arnedo,

Gallego-Durán,

Satorre-Cuerda,

Compañ-Rosique,

, & Molina-Carmona,

(2016). LudifyME: An Adaptive Learning Model Based on Gamification. In Xhafa

(Ed.), Formative Assessment, Learning Data Analytics and Gamification in ICT Education (pp. 245-269): Elsevier.

38.

Logan,

(2015). Deep Exploration of the Flipped Classroom before Implementing. Journal of Instructional Pedagogies, 16.

39.

losup,

, & Epema,

D. H.

(2014). An experience report on using gamification in technical higher education. Paper presented at the The ACM Special Interest Group on Computer Science Education (SIGCSE), Atlanta, Georgia, USA.

40.

Marshall,

Cardon,

, & Godin,

(2014). A Study of Project-Based Learning in an Introductory MIS Course. Issues in Information Systems, 15(2), 24-30.

41.

Mesquita,

Toda,

Brancher,

, & Do Carmo,

(2014). Utilizing gamification with social network to aid students in programming languages lessons in higher education IT courses. Paper presented at the Special Interest Group on Computer Science Education (SIGCSE), Hyatt Regency Atlanta.

42.

Mikkelsen,

T. R.

(2015). Nursing students’ experiences, perceptions and behavior in a flipped-classroom anatomy and physiology course. Journal of Nursing Education and Practice, 5(10), 28-35.

43.

Moore,

M. G.

(2016). Flipped classrooms, study centers andragogy and independent learning. American Journal of Distance Education, 30(2), 65-67.

44.

Oluwagbemi,

Misra,

, & Omoregbe,

(2014). Pervasive computing in classroom environments and applications. Paper presented at the Global Summit on Computer & Information Technology (GSCIT), Sousse, Tunisia.

45.

Orzan,

Iconaru,

, & Macovei,

O.-I.

(2012). An Empirical Model Of Online Buying Continuance Intention. Journal of Economic-Financial Theory and Practice, 3, 296-305.

46.

Poole,

S. M.

Kemp,

Patterson,

, & Williams,

(2014). Get your head in the game: Using gamification in business education to connect with Generation Y. Journal for Excellence in Business Education, 3(2).

47.

Rouibah,

(2012). Understanding student drivers and obstacles toward MIS major from the perspective of an arab country: The case of Kuwait. Issues in Information Systems, 13(2), 58-71.

48.

Sánchez-Mendiola,

Morales-Castillo,

Torruco-García,

, & Varela-Ruiz,

(2015). Eight years’ experience with a Medical Education Journal Club in Mexico: a quasi-experimental one-group study. BMC Medical Education, 15(222), 1-8.

49.

Santiago Jr,

J. M.

Guo,

Eng,

Kasley,

, & Phillips,

(2017). Introduction to Engineering Using Google Docs and Interactive Video in Support of an Online Flipped Classroom Approach. Paper presented at the American Society for Engineering Education (ASEE): Pacific Southwest Section, Tempe, Arizona, USA.

50.

Schultz,

Duffield,

Rasmussen,

S. C.

, & Wageman,

(2014). Effects of the flipped classroom model on student performance for advanced placement high school chemistry students. Journal of Chemical Education, 91(9), 1334-1339.

51.

Shroff,

R. H.

Deneen,

C. C.

, & Ng,

E. M.

(2011). Analysis of the technology acceptance model in examining students’ behavioural intention to use an e-portfolio system. Australasian Journal of Educational Technology, 27(4), 600-618.

52.

Sillaots,

(2015). Gamification of higher education by the example of computer games course. Paper presented at the The 7th International Conference on Mobile, Hybrid, and On-line Learning (eLmL), Lisbon, Portugal.

53.

Smith,

(2017). Gamified Modules for an Introductory Statistics Course and Their Impact on Attitudes and Learning. Simulation & Gaming, 48(6), 832-854.

54.

Surendeleg,

Murwa,

Yun,

H.-K.

, & Kim,

Y. S.

(2014). The role of gamification in education a literature review. Contemporary Engineering Sciences, 7(2932), 1609-1616.

55.

Tan,

Brainard,

, & Larkin,

G. L.

(2015). Acceptability of the flipped classroom approach for inäšhouse teaching in emergency medicine. Emergency Medicine Australasia, 27(5), 453-459.

56.

Tan,

, & Hew,

K. F.

(2016). Incorporating meaningful gamification in a blended learning research methods class: Examining student learning, engagement, and affective outcomes. Australasian Journal of Educational Technology, 32(5), 19-34.

57.

Tanner,

, & Scott,

(2015). A flipped classroom approach to teaching systems analysis, design and implementation. Journal of Information Technology Education (JITE): Research, 14, 219-241.

58.

Tolks,

Schäfer,

Raupach,

Kruse,

Sarikas,

Gerhardt-Szép,

, et al., (2016). An introduction to the inverted/flipped classroom model in education and advanced training in medicine and in the healthcare professions. GMS Journal for Medical Education, 33(3), 1-23.

59.

Veltsos,

J. R.

(2017). Gamification in the Business Communication Course. Business and Professional Communication Quarterly, 80(2), 194-216.

60.

Virkus,

(2015). Change and innovation in European library and information science education. BiD: Textos Universitaris de Biblioteconomia i Documentaci?, 35.

61.

Yang,

Asaad,

, & Dwivedi,

(2017). Examining the impact of gamification on intention of engagement and brand attitude in the marketing context. Computers in Human Behavior, 73, 459-469.

62.

Yildirim,

(2017). The effects of gamification-based teaching practices on student achievement and students’ attitudes toward lessons. The Internet and Higher Education, 33, 86-92.

63.

Zhonggen,

, & Guifang,

(2016). Academic Achievements and Satisfaction of the Clicker-Aided Flipped Business English Writing Class. Journal of Educational Technology & Society, 19(2), 298-312.