Effect of Digital Serious Games Related to Patient Care in Pharmacy Education: A Systematic Review

Abstract

Background. In recent years, the use of digital serious games in the education of healthcare students and professionals has been frequent. However, there are no high-level evidence studies focused on the effect of this tool in the development of patient care-related competencies in pharmacy education. Aim. To assess the effect of digital serious games on learning about patient care in pharmacy education. Methods. The Cochrane Library, ERIC, Embase, IPA, LILACS, PubMed, Scopus, and Web of Science databases were reviewed to identify relevant studies published up to October 2, 2018. Standardized and non-standardized terms including “games,” “serious games,” “pharmacy education,” “pharmacists,” and “pharmacy students” were used as search terms. The quality of the studies was assessed using validated tools. Results and Discussion. Of the 1,521 studies reviewed, seven met the eligibility criteria. Three studies were performed in the United States (42.85%). The most-frequent design studies were randomized controlled studies (n = 3; 42.85%). The number of participants ranged between 6 and 354. In most cases, game scenarios were based on simulations in which users performed the role of the pharmacist (n = 5; 71.42%). The predominant primary outcome was attitudes (n = 5; 71.42%) related to patient care. Only one study (14.27%) demonstrated significant improvements in the assessed primary outcomes. The methodology of the included studies lacked robustness. Conclusion. The effect of learning technological tools on patient care-related competencies was not significant in most of the included studies. Future studies must provide high-quality, integrated-manner evidence on the effect of digital serious games in pharmacy education (knowledge, skills, and attitudes) to facilitate the transfer of learning to real-life scenarios and changes in organizational practices.

Keywords

education learning < objective active learning < simulation/gaming educational games < simulation/gaming

Introduction

The changes that is occurring in the 21st century as the epidemiological and demographic transitions, new technological innovations, and populational demands as well as more complex and costly health systems require a new profile of healthcare professionals who present competencies related to critical thinking, problem-solving, and decision-making (Frenk et al., 2010). Thus, the use of active learning methodologies has become more important in the education of healthcare students and professionals. Active learning is grounded in self-directed and autonomous learning in which learners are involved in activities that promote analysis, synthesis, and evaluation (e.g., reading, writing, discussion, or problem solving) (Minhas et al., 2012; Vilela et al., 2013). Teachers take on the role of a facilitator or coach, whereas the learners assume ownership of their educational process (Uskokovic, 2017). In this context, the use of digital games as an active learning methodology is growing in recent years (Deshpande & Huang, 2011; Kostenius et al., 2018; Sangrà & González-Sanmamed, 2010).

Previous studies reported the use of digital games as a tool for active teaching and learning in health education (Andrews et al., 1992; Ellis & Hannigan, 1986; Payne & Rowberg, 1984). Since then, new digital games termed “serious” have been developed and utilized by students and health care professionals in the areas of Nursing, Pharmacy, and Medicine (Bernardo et al., 2018; Boada et al., 2018; Gallegos et al., 2017; Kleinert et al., 2015; Lam et al., 2019; Lee et al., 2018; Malhotra et al., 2017; Sera & Wheeler, 2017; Verkuyl et al., 2016). Serious games are games that are used in the teaching-learning processes or training, with the main purpose being to educate rather than to entertain (Baptista & Oliveira, 2018; Fonseca et al., 2014; Rodriguez et al., 2014; Serrano-Laguna et al., 2017; Tandogan & Orhan, 2007).

The “training simulation” (also called “game-based simulation” or “simulation game”) stands out among the types of digital serious games (Bigdeli & Kaufman, 2009; Graafland et al., 2012; Wang et al., 2016). Simulation is an instructional technique that mimics real clinical experiences by replicating essential aspects of a clinical situation in a safe environment (Aebersold, 2018; Gaba et al., 2004; Fowler-Durham & Alden, 2007; Hovancsek, 2007). Simulation-based education is used to develop competencies related to cognitive, psychomotor, and affective domains (Cant & Cooper, 2017; Cook et al., 2013; Kim et al., 2016; Nara et al., 2009; Warren et al., 2016). When serious games are used in simulation-based education, the learners need to seek new information (learning by discovery), explore previous knowledge (reflective learning), and make decisions as the scenarios change and the levels of difficulty increase (Gentry et al., 2019; Fox et al., 2018; Niemi, 2002). Also, its motivational effect has been demonstrated in a widely publicized study (Dominguez et al., 2013). Thus, digital serious games can help learners through self-directed, active, and motivational learning.

The use of digital serious games can allow the development of patient care-related knowledge, skills, and/or attitudes of health-care students (Barr, 2018; Gentry et al., 2019; Martí-Parreño et al., 2018; Sipiyaruk et al., 2018; Tandogan & Orhan, 2007). In the field of pharmacy, the use of digital serious games has increased because of the professional paradigmatic transition, with an emphasis on patient care that requires changes in the curricula and teaching strategies to pharmacy students and pharmacists (Anderson, 2002; Anderson et al., 2008a; Anderson et al., 2008b; Anderson et al., 2006; Audus et al., 2010; Awaisu & Mottram, 2018; Fathelrahman et al., 2016; International Pharmacy Federation, 2012; Lam et al., 2019; Mesquita et al., 2015; Toklu & Hussain, 2013; van Mil et al., 2005). Some systematic reviews are available on the use of digital serious games (played with an electronic device, e. g., computer or a mobile device) and non-digital serious games (so-called analog games or traditional games, e.g., card, board, or puzzle games) by students and professionals from different areas of Health Sciences, but no review has focused on patient care in pharmacy education (Bigdeli & Kaufman, 2009; Blakely et al., 2009; Capdarest-Arest et al., 2019; Gentry et al., 2019; Martins et al., 2015; Rienzo & Cubillos, 2020; Sardi et al., 2017; Tomé Filho et al., 2019; Wang et al., 2016).

This review aims to assess the effect of digital serious games on i) the development of knowledge, skills, and attitudes related to patient care in pharmacy education, ii) user satisfaction, economic outcomes, and unintended effects of the games, iii) the methodological quality of the studies.

Methods

This systematic review was performed using the Cochrane Handbook for Systematic Reviews of Interventions, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), and Assessing the Methodological Quality of Systematic Reviews (AMSTAR) (Cochrane, 2011; Moher et al., 2009; Shea et al., 2017). The protocol of this systematic review was registered in the PROSPERO database (CRD42019119306).

Primary and Secondary Outcomes

The primary outcomes of this review were the effects of digital serious games on the development of knowledge (e.g., disease and pharmacotherapy), skills (e.g., communication and clinical reasoning), and/or attitudes (e.g., behavior and motivation) related to patient care in pharmacy education. Secondary outcomes were related to user satisfaction with games, economic outcomes (e.g., cost-effectiveness), and unintended effects of the games (e.g., stress, fear, and anxiety).

Search in the Databases

Relevant studies published up to October 2, 2018, were identified through a search in the Cochrane Library, Education Resources Information Center (ERIC), Embase (Embase, MEDLINE, Embase Classic, PubMed-not-Medline), International Pharmaceutical Abstracts (IPA) (via ProQuest), Latin American and Caribbean Health Sciences Literature (LILACS) (via Virtual Health Library - VHL), PubMed, Scopus, and Web of Science. The search strategy employed standardized and nonstandard terms (text words) related to “games,” “serious games,” “pharmacy education,” “pharmacists,” and “pharmacy students” that were combined using Boolean operators (AND/OR). The standardized terms were selected using the controlled vocabulary of the “National Library of Medicines” through the “Medical Subject Headings” (MESH) (National Center for Biotechnology Information, 2018). The non-standardized terms were selected based on the study-subject related literature and used to extend the search strategy. No date limit was used in the database search. The complete search strategy is shown in Supplemental File 1.

Selection of the Studies

All duplicate studies were excluded. Two researchers (R.O.S.S. and A.M.P.) independently reviewed the titles and abstracts; full-texts were deemed relevant according to the eligibility criteria. Analysis of study duplicity as well as the selection of titles and abstracts was performed using the Rayyan tool (http://rayyan.qcri.org) (Ouzzani et al., 2016). Any divergence in study selection was judged by a third investigator (D.C.S.A.A.). For studies included after reading the full-text, manual screening of their references was performed to identify potentially relevant additional studies. For studies for which the full-text was unavailable, at least one author of the study was contacted via e-mail and/or ResearchGate (www.researchgate.net). The authors of this review also used the Bibliographic Commutation Program at the Federal University of Sergipe to obtain the full-texts.

Inclusion and Exclusion Criteria

Definitions for digital serious games, knowledge, skills, attitudes, and competencies were adopted and are presented in Supplemental File 2. Studies that met the following criteria were considered eligible for inclusion: 1) studies published in English, Spanish, or Portuguese; 2) used the term "game" to refer to the digital educational technology; 3) met the definition of a digital serious game that was adopted in this review; 4) applied the digital serious game with a focus on patient care in isolation or sets with other tools/methodologies to pharmacy students and/or pharmacists; 5) evaluated the effect of digital serious games on at least one of the primary outcomes established in this review. The exclusion criteria were as follows: 1) full-text was unavailable; 2) abstract published in conference proceedings; 3) letter to the editor; 4) literature review; 5) integrative review; 6) scoping review; 7) systematic review with or without meta-analysis; 8) overview of systematic reviews with or without meta-analysis; 9) book chapter; 10) dissertation; 11) thesis; 12) study in which the digital serious game was employed by students and/or professionals from different areas of Health Sciences, and it was not possible to extract data only for pharmacy students and/or pharmacists; 13) study in which the population and/or the intervention studied were unclear; 14) study in which the digital serious game was applied together with another tool/methodology and it was not possible to extract the results related to the digital serious game. The year of publication and the methodological quality were not used as exclusion criteria.

Data Extraction

Two reviewers (R.O.S.S. and A.M.P.) independently extracted the data from the included articles using a spreadsheet pre-formatted in Microsoft® Excel®. The extracted data were as follows: authors, year of publication, study objective, country, design, duration, number and type of participants, year of the graduation, discipline, and the type of discipline (mandatory or elective) in studies in which the digital serious game was applied to pharmacy students; characteristics of the digital serious game; definitions adopted for knowledge, skills, attitudes, game and/or serious game; primary and/or secondary outcomes assessed; methods for assessing primary and/or secondary outcomes; effect of the digital serious games on the primary and secondary outcomes (main results); stated limitations or bias. Any divergence in the data extraction was resolved by reaching a consensus.

Assessment of Methodological Euality

The methodological quality of the included qualitative studies was assessed using the Critical Appraisal Skills Program (CASP) Qualitative Research Checklist, for qualitative studies; and the Medical Education Research Study Quality Instrument (MERSQI) was used for studies with other designs (experimental, quasi-experimental, and observational studies). The CASP consists of ten questions divided into three domains and can be answered as “yes,” “no” or “not reported” (Critical Appraisal Skills Programme, 2018; Kim et al., 2017). The MERSQI presents ten items that are divided into six domains. The score of this tool can range from 5 to 18, with the highest score indicating the best methodological quality (Gorbanev et al., 2018; Reed et al., 2007; Sullivan, 2011). The quality of the included studies was independently assessed by two researchers (K.S.S.R. and D.C.S.A.A.) and the discrepancies were resolved by reaching a consensus.

Results

Study Selection

The bibliographic search identified 1,521 studies; 1,449 were excluded after analyzing for duplicates and reading titles and abstracts. Then, 72 studies were selected for full-text reading; seven of the studies met the inclusion criteria (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Kinkade et al., 1995; Lambertsen et al., 2016; Smith et al., 2016; Smith et al., 2017). The references included in these studies were manually reviewed and 16 potentially relevant studies were identified. However, none met the inclusion criteria. Thus, at the end of the selection screening, seven studies were included. The studies identified by searching the databases that were excluded after full-text review together with the reasons for exclusions are summarized in Supplemental File 3. Figure 1 illustrates the study-selection process.

Study Characteristics

The characteristics of the included studies are described in Table 1. All articles were published in English between 1995 and 2018. These studies were performed in America, Europe, and Oceania. Three studies (42.85%) were performed in the United States of America (Kinkade et al., 1995; Smith et al., 2016; Smith et al., 2017). The most-frequent design studies were randomized controlled studies (n = 3; 42.85%) (Berger et al., 2018; Bindoff et al., 2014; Kinkade et al., 1995) and before-and-after studies without a control group (n = 2; 28.57%) (Smith et al., 2016; Smith et al., 2017). Furthermore, a majority of the studies (n = 6; 85.71%) failed to report the duration of the study (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Kinkade et al., 1995; Lambertsen et al., 2016; Smith et al., 2016). The only study that reported this variable took nine months (Smith et al., 2017).

Table 1.

Characteristics of the included studies in the systematic review

Reference	Country	Objective	Design	Duration	Population
Berger et al. (2018)	Switzerland and Belgium	To determine whether participation rate and students’ post-experiment statements and knowledge were higher in the digital serious game group compared to the paper-based group and to determine the differences observed between the four subgroups: Belgian students – who had no lectures on the theme of the clinical cases – in the game and paper-based group; and, Swiss students – who had such lectures – in the game and paper-based group)	Open double-center, randomized and parallel-group(randomized controlled study¹)	NR	70 MSc students in Pharmacy (second-year students; 43 Belgians and 27 Swiss; 41 in the digital serious game group and 29 in the paper-based group)
Bindoff et al. (2014)	Australia	To provide a digital serious game-based learning method for pharmacy practice that is as effective as paper-based scenarios, but more engaging and less labor-intensive	Randomized controlled study	NR	3 PhD students in Pharmacy³ and33 B Pharm students (third-year and fourth-year students; 16 in the digital serious game group and 17 in the paper-based group)
Hookham et al. (2015)	Australia	To report the first phase of formative assessment, observing the effectiveness and perceived usability of a digital serious game	NR(intervention study without control group with post-test¹)	NR	10 MSc students in Pharmacy
Kinkade et al. (1995)	USA	To develop a digital serious game about glaucoma and test whether the simulation was equivalent to the standard case study on the same subject	NR(randomized controlled study¹)	NR	48 Pharmacy students(24 in the digital serious game group and 24 in the paper-based group)
Lambertsen et al. (2016)	UK	To assess whether digital serious game could be used as a learning tool in pharmaceutical education at the university level	NR(mixed design: direct observation of participants and qualitative study^{1, 2})	NR	6 Pharmacy students (level 6)
Smith et al. (2016)	USA	To measure the attitudes of undergraduate health students towards poverty before and after playing the SPENT game	NR(quasi-experimental study – before-and-after study without a control group¹)	NR	354 health professions (Pharmacy students - second preprofessional year and first professional year; physician assistant, and communication science and disorders), of which 240 were Pharmacy students
Smith et. (2017)	USA	To identify whether the SPENT game could improve students' attitudes towards those living in poverty and to determine whether changes resulting from the use of SPENT have been maintained for months	Quasi-experimental post hoc analysis(quasi-experimental study – before-and-after study without a control group¹)	9 months	99 Pharmacy students (first-year and second-year professional phase)

¹ Design was defined by the authors of this systematic review.

² Only data regarding the qualitative methodology were extracted.

³ The game was pilot tested by 3 PhD students to ensure it worked correctly.

Abbreviations: B Pharm – Bachelor of Pharmacy; CAPS - Community Action Poverty Simulation; MSc – Master of Science; UK - United Kingdom USA - United States of America; NR- Not reported.

The number of included participants ranged from 6 to 354 (Lambertsen et al., 2016; Smith et al., 2016). The population consisted of students in different stage of learning in the field of pharmacy (e.g., preprofessional year, professional year, B Pharm, MSc, PhD). One study included participants other than pharmacy students (Smith et al., 2016). A majority of the studies (n = 5; 71.42%) did not report the disciplines to which the games were applied (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Lambertsen et al., 2016; Smith et al., 2016). In the case of studies reporting this variable, one study applied the game to the discipline of “Pathophysiology and Pharmacotherapy” (Kinkade et al., 1995) and another study to “Provision of Health Care” (Smith et al., 2017). No study defined whether these disciplines were mandatory or elective in the pharmaceutical curriculum.

Definitions, terminologies, Characteristics, Duration of the Application of Digital Serious Games and Characteristics of the Control Group

The definitions of the games and/or serious games, the terminologies used to refer to digital serious games, the characteristics of digital serious games, and the tools and methodologies used in the control group are presented in Table 2. Most studies (n = 5; 71.42%) failed to present definitions for serious games or games (Bindoff et al., 2014; Kinkade et al., 1995; Lambertsen et al., 2016; Smith et al., 2016; Smith et al., 2017).

Table 2.

Terminologies adopted for digital serious games, definitions adopted for games and/or serious games, characteristics of the digital serious games, and characteristics of the tools/methodologies used in the control group

Reference	Terminologies	Definitions	Characteristics of the digital serious games	Characteristics of the control group
Berger et al. (2018)	Serious game	A digital serious game is “a mental contest, played with a computer following specific rules, that uses entertainment, to furthergovernment or corporate training, education, health, public policy, and strategic communication objectives”	Simulation of a community pharmacy environment where the user provides care to the father of a two-year-old patient who requests a cough syrup (contraindicated in this case). The information to resolve the case is given by the father's avatar, in response to the pharmacist (simulating an interview). In the end, users receive feedback on the chosen alternatives	Paper-based clinical cases equal to the intervention group. The students followed a linear and identical path, regardless of the answers. They received feedback before the next step and chose correct answers at each step (corresponding to the questions/actions they would do)
Bindoff et al. (2014)	Game, software, computer simulation, computer-based simulation, and virtual simulation	NR	Virtual Dispensary exposes users to two cases in a community pharmacy: (1) patient with heart failure with a new prescription from the general practitioner and (2) patient with the same clinical condition and with back pain (for which he has a medicine prescription contraindicated). The virtual scenario has a reception, dispensary, computer, and telephones. Conversations with the patient are divided by opening questions/greetings, history taking, recommendations, and advice. The patient responds to the user's actions (e.g. shaking his head). Users can also call the prescriber, the hospital, or a nearby pharmacy. After completing the case, the user receives feedback and a score	Paper-based clinical cases equal to the intervention group. This group had access to two facilitators with clinical knowledge with whom they could discuss the scenarios
Hookham et al. (2015)	Game and serious game	“The characteristics of a game are a goal, rules, feedback, andvoluntary participation A goal motivates the player; the rules define how that player can approach the goal; feedback informs the player of their performance, and voluntary participation implies the choice the player makes to enter the game world and interact with that world”	Scenario consisting of 16 images of a 360°environment of a community pharmacy. Information about pharmacy locations is provided through auditory descriptions and information pop-ups. A single session requires the user to access a quiz, which gives him access to the Virtual Dispensary with two main areas: one where the patient submits a prescription to the pharmacist (Scripts In) and another one where the pharmacist ensures that prescriptions, medicines, and labels are correct and appropriate (Script Checking). Prescriptions and medicines are displayed in an interactive 3D model	NA
Kinkade et al. (1995)	Game, program, computer program, computer simulation and computer-assisted patient simulation	NR	Clinical case of a patient with open-angle glaucoma. The virtual environment has two rooms: the reference room (it shows the patient, intraocular pressure, and list of medicines available for use) and the monitoring room (it provides new complaints related to intraocular pressure and side effects of the medicines)	Paper-based clinical case like the intervention group
Lambertsen et al. (2016)	Game, serious game, videogame, virtual patient, and virtual patient simulator	NR	PharmaComm is a simulation of a hospital environment that begins with the patient admission. The user must decide the medicine administration and doses. For this, he receives information about the patient's clinical situation. When the user acts, the virtual patient reacts accordingly (e.g. decrease/increase in heart rate). In the end, feedback is presented to the user	NA
Smith et al. (2016)	Game, online game, computer simulation game, and online simulation	NR	SPENT requires users to survive on US$1000 for one month. The player must make decisions (e.g. to select a place to live and to purchase health insurance) and, for each of them, he/she receives a potential result of how this would affect an individual living in poverty. The game ends if the player runs out of money or completes the month with money	NA
Smith et. (2017)	Game, online game, online serious game, and computer game	NR	SPENT ¹	NA

¹ This game is also used in the study by Smith and colleagues (2017). Therefore, the game characteristics are the same.

Abbreviations: NA - Not Applicable; NR - Not Reported.

The studies that presented this definition agreed that games are systems having rules that define how the user reaches the objective of the game (Berger et al., 2018; Hookham et al., 2015). As for the terminologies used to refer to the evaluated intervention, the most frequent terms were serious game (n = 4; 57.14%) (Berger et al., 2018; Hookham et al., 2015; Lambertsen et al., 2016; Smith et al., 2017) and computer simulation (or computer-based) (n = 3; 42.85%) (Bindoff et al., 2014; Kinkade et al., 1995; Smith et al., 2016).

In most studies (n = 5; 71.42%), the game scenarios were based on simulations in which users represented the pharmacist’s role (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Kinkade et al., 1995; Lambertsen et al., 2016). The most frequent scenario was community pharmacy (n = 3; 42.85%) (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015). Furthermore, in three studies (42.85%), the games used elements related to the practice of drug dispensing (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015). Two studies used the same game (SPENT) (Smith et al., 2016; Smith et al., 2017). The freedom to make decisions was a feature present in most of the evaluated games, implying that there is no single way to solve the game (Berger et al., 2018; Bindoff et al., 2014; Lambertsen et al., 2016; Smith et al., 2017). These games allowed different “learning paths” with individualized experiences among students.

The lowest average duration of the games was 30 min (Hookham et al., 2015). However, three studies (42.85%) had an average duration of 50 min (Lambertsen et al., 2016; Smith et al., 2016; Smith et al., 2017). Two studies (28.57%) failed to report this variable (Berger et al., 2018; Kinkade et al., 1995). Regarding the characteristics of the interventions performed in the control group, three studies presented paper-based clinical cases that were similar or equal to the cases used in the games (Berger et al., 2018; Bindoff et al., 2014; Kinkade et al., 1995).

Outcomes, Evaluation Methods, Main Results, and Limitations

The outcomes, assessment methods, and main results of the studies are described in Table 3. The most frequent primary outcomes were related to knowledge (n = 3; 42.85%) (Berger et al., 2018; Bindoff et al., 2014; Kinkade et al., 1995) and attitudes (n = 5; 71.42%) (Berger et al., 2018; Hookham et al., 2015; Lambertsen et al., 2016; Smith et al., 2016; Smith et al., 2017). Only one study evaluated the participant’s perceptions on skills (Berger et al., 2018). No study presented definitions of knowledge, skills, and attitudes.

Table 3.

Primary and/or secondary outcomes, methods for outcomes assessment and main results

Reference	Primary and/or secondary outcomes¹	Methods for outcome assessment	Main results
Berger et al. (2018)	-Primary outcomes:Behavioral theory related to pharmaceutical screening; self-reported skills and knowledge for cough triage -Secondary outcomes:usability and satisfaction with the game	Pre- and post-intervention assessment using a 6-point Likert scale on opinions about pharmaceutical screening, skills, and knowledge for cough triage	-Primary outcomes⁴- The theory of planned behavior showed a difference only for the statement “Pharmacist triage is a good way to manage patients’ health problems”: 0,0 ± 0,8 vs 0,5 ± 1,3 (p = 0,02)- Increased confidence in the skills to conduct proper triage even in complex situations for both groups- There was no difference in skills and knowledge for cough triage- There was no difference in clinical knowledge scores: 7,07 ± 1,50 vs 6,81 ± 1,10 (p = 0,27) -Secondary outcomes⁴- The students in the game group found it more difficult to navigate the case: 2,4 ± 1,6 vs 3,3 ± 1,7 (p = 0,03)- The students in the intervention group felt more often that they did not simulate a real situation: 2,4 ± 1,5 vs 1,8 ± 1,1 (p = 0,05)
Bindoff et al. (2014)	- Primary outcomes:knowledge of the treatment of heart failure; perception of anamnesis and counseling skills -Secondary outcomes:usability of the game	-Pre- and post-intervention in the intervention and control group: multiple-choice test on the treatment of heart failure and a self-assessment survey on anamnesis and counseling skills - Post-intervention in the intervention group: evaluation of the game's usability using a 5-point Likert scale and feedback on how it could be improved in the future	- Primary outcomes ⁴-Clinical knowledge did not improve significantly after the game (F (1, 29) = 0.63, p = 0.433), but the difference between the scores of the groups approximated the significance (F (1, 29) = 3.85, p = 0.059)- The intervention group showed an apparent improvement in the clinical knowledge and a significantly improved perception about history taking (p<0,03) and counseling competencies (p<0,08) when compared to the third-year control group -Secondary outcomes: the most common comments regarding usability were difficulty using the system, overly simplistic dialogue options, and the value of being able to perform the entire dispensing process
Hookham et al. (2015)	- Primary outcome:self-reported attitudes about dispensing skills- Secondary outcomes:usability and satisfaction with the game	Post-intervention: questions answered according to a 5-point Likert scale	- Primary outcome: All participants agreed that the game provided an effective way to learn about dispensing- Most participants agreed that the game improved their confidence about the dispensing- Secondary outcomes: 7 participants agreed that the game was easy to use, 9 that it was fun, 10 that the world was realistic, and 8 felt they could learn to use the game quickly-There were divergent opinions about the need for further instructions to use the game- Suggested improvements: an increase in the number of products and scripts available, more information about the product to be integrated into the game, and greater integration of elements of pharmaceutical practice, such as patient counseling
Kinkade et al. (1995)	Primary outcomes:knowledge about Pathophysiology and Pharmacotherapeutics	Post-intervention: test with multiple- choice questions and true or false questions	- Test F showed that there were no significant differences between the two groups 8,21± 1,21 vs 7,83 ± 1,93 ⁴- A subsequent calculation for the effect size showed that there was a small effect (0,31) ⁴
Lambertsen et al. (2016)	- Primary outcomes:self-reported attitude about communication skills and medication administration- Secondary outcomes:usability and satisfaction with the game	Direct observation of participants and qualitative study ²	- Primary outcomes: Students reported that they could practice their skills without being afraid of making mistakes, being able to increase their confidence- Secondary outcomes: All participants liked the fact that they get instant feedback on their actions through vital signs; that the animation made the tool look realistic. Participants agreed that it was fun to use the game and that, although the game was educational, it did not look like they were learning while using it
Smith et al. (2016)	Primary outcome:attitudes toward people in poverty	Pre- and post-intervention: UPPTS instrument with statements about the condition of poverty assessed using a Likert scale, whose scores ranged from 30 to 195. Higher scores suggested a more empathic attitude towards people living in poverty	- Pharmacy students in the pre-professional phase: 129.2 vs 124.6 ⁶- Pharmacy students in the professional phase: 133.2 vs 135.4⁶- General average score of attitudes: 132.1 vs 136.2 (t305 = -8.03, p <0.0005) ^{5, 6}- Students with the lowest attitudes improved even more when compared to the others: 108.8 vs 114.70 (t76, p> 0.0005) ^{5, 6}
Smith et. (2017)	- Primary outcomes:attitudes toward people in poverty, knowledge of resources available to patients in economic difficulties, and understanding of decisions that a person living in poverty can make regarding health care - Secondary outcome:satisfaction with the game	- Pre- and post-intervention: UPPTS instrument ³ - Post-intervention: questions about resources available to patients in economic difficulties and about decisions that a person living in poverty can make regarding health care. Users provided comments on how to help people living in poverty	- Primary outcomes: General average score of attitudes: 133,52 ± 17,35 vs 135,54 ± 18,75 (p = 0,046)- Knowledge of resources available to patients in economic difficulties and understanding of decisions that a person living in poverty can make regarding health care, the majority students responded that: “Living in poverty is emotionally hard and stressful”; “I learned about the difficult choices individuals who live in poverty have to make”; and, "I understand why a person may decide not to purchase insurance" - Secondary outcome: 50.5% (n = 50) of the students felt that the game was worthwhile, 28.3% (n = 28) that it was very worthwhile, and 2% (n = 2) reported that the game was not worthwhile

¹ The primary and secondary outcomes were defined according to the outcomes of interest in this systematic review.

² Only data regarding the qualitative methodology were extracted.

³ This instrument is also used in the study by Smith and colleagues (2017). Therefore, the instrument characteristics are the same.

⁴ Results of intervention vs control group.

⁵ Results of pre- vs post-intervention.

⁶ The analysis of statistical significance was performed for all study participants, which includes other participants besides Pharmacy students.

Abbreviations: NR - Not Reported; UPPTS - Undergraduate Perceptions of Poverty Tracking Survey.

Tests and questionnaires for self-evaluation were the most frequent methods for assessing knowledge (Bindoff et al., 2014; Kinkade et al., 1995), skills, and attitudes (Berger et al., 2018; Hookham et al., 2015; Smith et al., 2016; Smith et al., 2017). When comparing the baseline results with the post-intervention results of the two before-and-after studies without a control group (n = 2; 28.57%) (Smith et al., 2016; Smith et al., 2017), a significant improvement was noted in one study (Smith et al., 2017).

In the study by Smith et al. (2016), there was an improvement in the results when comparing the baseline results with the post-intervention results. However, it was not possible to confirm whether it was a significant improvement since the analysis of statistical significance was performed for all study participants, and not just for pharmacy students. As for the studies that presented a control group (n = 3; 42.85%) (Berger et al., 2018; Bindoff et al., 2014; Kinkade et al., 1995), no significant improvement was observed in the intervention group compared to the control group.

Usability and satisfaction with games were the most evaluated secondary outcomes (n = 5; 71.42%) (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Lambertsen et al., 2016; Smith et al., 2017). These variables were evaluated positively in only two studies (28.57%) (Lambertsen et al., 2016; Smith et al., 2017). The most cited limitations were related to sampling characteristics (e.g., a small number of participants, the predominance of a certain gender, and differences in the number of participants between groups) (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015; Smith et al., 2017) and design (e.g., absence of a control group) (Smith et al., 2016; Smith et al., 2017). One study did not report limitations (Lambertsen et al., 2016).

Quality Assessment

Only one study was evaluated using the CASP tool (Supplemental File 4) (Lambertsen et al., 2016). This study presented a clear description of the research objectives and an adequate recruitment strategy, clearly demonstrating the significance of the research. However, most of the items received an answer “no” (n = 4; 40%) or “not reported” (n = 3; 30%), indicating gaps in their methodological quality. Other studies were evaluated using the MERSQI tool (n = 6) (Supplemental File 5) (Bindoff et al., 2014; Hookham et al., 2015; Kinkade et al., 1995; Smith et al., 2016; Smith et al., 2017). The score ranged from 6.0 to 12.5, with an average of 9.83 ± 2.2 (Berger et al., 2018; Bindoff et al., 2014; Hookham et al., 2015). The scores were higher for the data-analysis domain. No article presented tests of internal validity or relevant evidence regarding relationships with other variables.

Discussion

Serious games are an educational strategy widely used in areas such as medicine and nursing (Blakely et al., 2009; Gentry et al., 2019; Graafland et al., 2012). However, this systematic review found only a small number of studies in the field of pharmacy. A factor that may have influenced this number was the focus to apply digital serious games to patient care. It is worth mentioning that there are studies on digital serious games in other areas of pharmacy education (such as drug production) (Cruz-Benito et al., 2016; Dudzinski et al., 2013; Ee et al., 2018; Wang et al., 2016) that were excluded from this review. Thus, given its purpose, the number of included studies may not reflect the use of digital serious games in other areas of pharmacy education.

The American Association of Colleges of Pharmacy and Accreditation Council for Pharmacy Education encourage the use of technologies as a complementary teaching tool. This may explain the high number of studies performed in the United States that were included in this systematic review (Accreditation Council for Pharmacy Education, 2015; Cain et al., 2014; Crass & Romanelli, 2018). Other factors that may reflect this number are the pioneering role of the United States in pharmaceutical care and the use of digital serious games in other areas such as economics, military, and healthcare as well as the ability to incur the high cost involved in the development of these technologies (Carter, 2016; Gentry et al., 2019; Ijaz et al., 2019; Sardi et al., 2017).

Similar to our systematic review, some reviews identified a high number of publications on gamified apps and serious games in the healthcare field that became available after 2014 (Gentry et al., 2019; Gorbanev et al., 2018; Wang et al., 2016). This likely reflects the recent trend to use digital serious games in the teaching of pharmacy. More recently, the Report of the Academic Affairs Committee 2013 - 2014 of the American Association of Colleges of Pharmacy declared support for the use and development of serious games to improve pharmaceutical and interprofessional education (American Association of Colleges of Pharmacy, 2013; Cain et al., 2014).

In the included studies, a terminology frequently used for digital serious games was computer (or computer-based") simulation. Thus, it is necessary to discuss when a simulation becomes a game. According to Wang et al. (2016), there is no clear concept or consensus that differentiates simulations and games. These authors argue, for example, that meetings with virtual patients are not necessarily digital serious games solely owing to the use of computers. Bigdeli and Kaufman (2017) determine that the terms “game”, “simulation” and “simulated game” are used interchangeably with overlapping definitions. For these authors, games and simulations have similarities as rules, mechanics, collaboration, and interactivity with high reasoning and mental engagement. On the contrary, games have elements of competition and entertainment, while simulations try to mimic real-life and normally do not possess these elements.

In this context, we understand that simulation and digital serious games are different terminologies that can be related. Simulation is an active learning methodology that can be performed using different techniques and tools, such as role-play, simulated patients, mannequins, and digital technological tools (Curtin et al., 2011; Issenberg & Scalese, 2008; Lins et al., 2011; Noori et al., 2014; Vyas et al., 2013). Digital serious games, in turn, are digital educational technologies that have different genres such as adaptation, adventure, board, puzzle, quiz, and simulation (Bigdeli & Kaufman, 2009; Graafland et al., 2012; Wang et al., 2016). Thus, we believe that not every digital serious game can be considered a simulation, and not all digital simulation occurs through digital serious games. The definition of a digital serious game as a technological tool for simulation will depend on its characteristics. In this sense, we intend this review to be a new starting point for discussions related to the definitions of “game”, “simulation,” and “simulated game” in the literature.

As for the characteristics of digital serious games, a majority of studies included in this review used games to develop knowledge and attitudes related to drug dispensing. This is a clinical pharmacy service that provides medicines or health products through the analysis of technical and legal aspects of prescription, assessment of patients’ individual health needs, the performance of interventions in the process of medication use, counseling, and documentation of the interventions) in community pharmacies (Ministry of Health of Brazil, 1998; Hernández et al., 2015; National Health Service, 2013). Drug dispensing is the most traditional clinical service performed by pharmacists (Hawksworth et al., 1999; Holland & Nimmo, 1999). Furthermore, studies have shown that drug dispensing is the most widely disseminated service in community pharmacies around the world that serves a variety of people and requires pharmacists to have knowledge, skills, and attitudes to provide quality care (Abaurre et al., 2014; Angonesi & Renno, 2011; Martins et al., 2015). In drug dispensing, as in other pharmaceutical clinical services, medication is an important instrument but the focus of the service must be on the patient’s needs. Therefore, there is a need for educational tools that contribute to the development of the competencies related to drug dispensing (i.e., patient education and counseling) in pharmacy education.

The assessment of attitudes and behavioral aspects was performed in most of the included studies in this systematic review. In contrast, a systematic review of the use of games in medical education identified that most articles focused on assessing only knowledge and skills (Gorbanev et al., 2018). Another noteworthy finding was that no article included in our review assessed the effect of games on the development of competencies in an integrated manner (knowledge, skills, and attitudes) in pharmacy education when related to patient care. Future studies should focus on all competencies to train motivated and qualified professionals to improve patient care.

The assessment of competencies related to patient care, especially when performed in an integrated manner, can identify gaps between teaching and learning (Croft et al., 2019; Hill et al., 2006). Consequently, they can help to identify needs and generate new learning approaches for developing competencies needed to provide quality pharmaceutical care. From this perspective, most studies included in this review did not demonstrate a significant improvement in knowledge, skills, and attitudes. In contrast, a systematic review that evaluated the use of serious games in the training of health professionals showed that, of the 19 studies, only two did not report significant differences in knowledge and skills between the control and intervention groups (Wang et al., 2016).

In terms of the effectiveness of digital serious games, the differences observed between this review and other studies in the literature may be related to the methodological quality of the reviewed studies. The included studies had an average score below the values observed in other systematic reviews (Gorbanev et al., 2018; Wang et al., 2016). Moreover, most included studies did not report their duration and variabilities were observed in the sample type and size. Another issue that is very important for the effectiveness of the games used as tools is validation. The studies included in this review failed to present information on the previous validation of games before their use as teaching tools. According to the literature, serious games should go through a validation process that includes tests of apparent validity: content, construct, concurrent, discriminant, and predictive (Gallagher et al., 2003; Schijven & Jakimowicz, 2005). These findings about duration, sample type and size, as well as validation can compromise the evaluation of digital serious game effects, once those characteristics are essential for any scientific research as they allow the generation of robust evidence regarding the effect of these tools on pharmacy education.

Most of the studies included in this review produced negative results related to the usability and satisfaction of digital serious games. Systematic reviews evaluated the satisfaction of healthcare students and professionals with serious games in a non-grouped manner and, consequently, in an inconclusive manner (Gorbanev et al., 2018; Ijaz et al., 2019; Wang et al., 2016). Motivation and satisfaction are important elements to achieve the proposed learning objectives since they can influence the users’ decision to use the game (Liao & Wang, 2011; Liao, 2015, Vlachopoulos & Makri, 2017). It is also important to emphasize that learning satisfaction can be related to users’ intrinsic factors as well as characteristics of the games and their application (Mayer, 2013). Thus, another hypothesis to explain the negative results related to the effectiveness of serious games in pharmacy education might be the relationship between usability, satisfaction, and learning outcomes. That is, the negative results of these tools related to usability may have influenced satisfaction and, consequently, the learning results. Therefore, future studies may appropriate the findings of this review and give greater emphasis on usability in the development of digital serious games, as well as assessing the relationship between usability of games, satisfaction, and learning outcomes.

Given the data presented in our review, limited and uncertain scientific evidence is available on the effectiveness of serious games on the development of knowledge, skills, and attitudes related to patient care in pharmacy education. However, literature has reported that the main advantage of digital serious games is the involvement and motivation of users in the teaching-learning process (Graafland et al., 2012). This is important since the apprentices of the current generation of “digital natives” grew up exposed to technologies and therefore have educational needs different from those that would be complementary to traditional methods (Inter-American Development Bank, 2006).

To overcome these challenges, educational institutions must invest in the development and acquisition of technological resources such as serious games to stimulate the teaching-learning process. Ultimately, stimulating games should be played for pleasure and not simply to fulfill a curricular requirement (Ravyse et al., 2017). Despite the complexity involved, following the stages of serious game development recommended in the literature as well as the validation process remains essential for the development of quality and stimulating tools, ensuring that educational goals are achieved. Therefore, educational institutions must be concerned with the effect of digital serious games on transferring learning to real practice scenarios and with changes in organizational practices leading to better results for patients.

Directions for Assuring the Quality of Future Studies Applying Digital Serious Games in Pharmacy Education

The heterogeneity, methodological quality, and the number of included studies, while not being issues inherent to the methodology of this systematic review, may have compromised its main objective. The heterogeneity of studies in the learners’ stage of study (undergraduate or graduate) and the assessment methods may have influenced the comparability and generalization of the findings. The methodological flaws in the included studies such as the failure to mention the duration of the study, use of small samples, and the low number of validated games may have compromised data extraction as well as the generation of robust evidence on the effect of digital serious games on teaching the topic of patient care in pharmacy education. These issues must be overcome in future studies on this topic.

The quality assessment of the included studies has identified other methodological issues to be addressed in future research. Specifically, experimental, quasi-experimental, and observational studies should focus mainly on the following aspects: to conduct studies in more than one institution; to assess the effectiveness of digital serious games using objective measurement; to assess the validity of the evaluation instrument, present the internal structure, content, and relationships to other variables; to present data analysis appropriate for study design or type of data; to conduct complex (not just a descriptive) data analysis. Qualitative studies should focus mainly on the following issues: to assess if the qualitative methodology is appropriate for study; to justify the research design; to collect the data in a way that addresses the research issue; to examine the relationship between the researcher and participants; to present the ethical issues; rigorously to analyze data; to present a clear statement of findings.

Strengths and Limitations

This study has several strengths and limitations. To our knowledge, this is the first systematic review on the effect of digital serious games related to patient care in pharmacy education. This review did not limit the selection of the year of publication and methodological quality of the studies. It included both qualitative studies and intervention studies, enabling a broader view of the topic and identifying the methodological gaps that must be filled in future studies. Also, a rigorous method of data compilation and presentation was used by two reviewers evaluating titles, abstracts, and full-texts as well as data extraction. The same reviewers also performed quality assessment using the tools recommended by the literature that increased the internal validity of the study findings. This review followed the Cochrane guidelines for systematic review as well as the AMSTAR and PRISMA tools that supported the review quality. Conversely, the absence of a search in the gray literature and the inclusion of studies that used the term "game" to refer to the digital educational technology could be considered as limitations that may have influenced the number of studies included in this review.

Conclusions

The assessment of the effectiveness of digital serious games for patient care in pharmacy education is primarily performed in a fragmented manner with a focus on knowledge, skills, or attitudes. In this review, a majority of the included studies found no significant improvement in the primary outcomes. Also, evidence on this subject is limited and not very robust due to the low methodological quality and number of studies on the effect of these learning tools on developing competencies relevant to patient care in pharmacy education.

Future studies must provide high-quality, evaluating the effect of digital serious games in an integrated manner, allowing the transfer of learning to real-life scenarios and facilitating changes in organizational practices of pharmacy students and pharmacists who provide patient care.

Footnotes

Acknowledgments

The authors thanks specially to Alfredo Dias de Oliveira Filho, Alessandra Rezende Mesquita, Daniel Tenório da Silva, Giselle de Carvalho Brito, Izadora Menezes da Cunha Barros, Sabrina Cerqueira Santos, as well as, the researchers of the Laboratory of Teaching and Research in Social Pharmacy (LEPFS) for their contributions to this manuscript.

Declaration of Conflicting Interests

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior --Brasil (CAPES) --Finance Code 001. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. In addition, the authors (ROSS, DCSAA and KSSR) received financial support of the Coordination of Superior Level Staff Improvement (CAPES) and Technological Innovation Support of the State of Sergipe (FAPITEC/SE) (scholarship/research grant).

ORCID iD

Divaldo Pereira de Lyra Júnior

Author Biographies

Rafaella de Oliveira Santos Silva is a PhD student in the Pharmaceutical Sciences Graduate Program at the Federal University of Sergipe (UFS), Brazil. She is researcher at Laboratory of Teaching and Research in Social Pharmacy (LEPFS), and her research interests include clinical pharmacy services, pharmacy education and educational technology.

Contact: rafaellaa_oliveira@hotmail.com

André Mascarenhas Pereira is pharmacist and researcher at Laboratory of Teaching and Research in Social Pharmacy (LEPFS) at the Federal University of Sergipe, Brazil. His research interests include pharmacy education.

Contact: pharm.andrepereira@gmail.com

Dyego Carlos Souza Anacleto de Araújo is a PhD student in the Health Sciences Graduate Program at the Federal University of Sergipe (UFS), Brazil. He is researcher at Laboratory of Teaching and Research in Social Pharmacy (LEPFS), and his research interests include clinical communication skills and pharmacy education.

Contact: dyegodm_pb@hotmail.com

Kérilin Stancine Santos Rocha is a PhD student in the Health Sciences Graduate Program at the Federal University of Sergipe (UFS), Brazil. She is researcher at Laboratory of Teaching and Research in Social Pharmacy (LEPFS), and her research interests include clinical pharmacy services, implementation science, evidence-based health, and pharmacy education.

Contact: kerilin.farm@gmail.com

Mairim Russo Serafini is a professor in the Department of Pharmacy, as well in Pharmaceutical Sciences and Health Sciences Graduate Programs at the Federal University of Sergipe (UFS), Brazil. Her research interests focuses on pharmaceutical technology.

Contact: maiserafini@hotmail.com

Divaldo Pereira de Lyra Jr. is a professor in the Department of Pharmacy, as well as Pharmaceutical Sciences and Health Sciences Graduate Programs at the Federal University of Sergipe (UFS), Brazil. In addition, he is leader of Laboratory of Teaching and Research in Social Pharmacy (LEPFS). His research interests include clinical pharmacy services, implementation science, pharmacy education, and educational technology.

Contact: lepfs.ufs@hotmail.com

References

Abaurre

, García-Delgado

, Maurandi

M.D.

, Arrebola

, Gastelurrutia

M.Á.

, & Martínez-Martínez

(2014). Diseño y pilotaje de un proceso estructurado para el servicio de dispensación de medicamentos. Atención Primaria , 47(1), 7–14. https://doi.org/10.1016/j.aprim.2014.01.005.

Accreditation Council for Pharmacy Education. (2015). Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. (Standards 2016). Accreditation Council for Pharmacy Education https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf.

Aebersold

(2018). Simulation-based learning: No longer a novelty in undergraduate education. The Online Journal of Issues in Nursing , 23 (2). https://doi.org/10.3912/OJIN.Vol23No02PPT39.

American Association of Colleges of Pharmacy. 2013. 2014 American Association of Colleges of Pharmacy annual report. American Journal of Pharmaceutical Education , 78(8), S4. https://doi.org/10.5688/ajpe788S4.

Anderson

, Bates

, Beck

, Brock

, Futter

, Mercer

, Rouse

, Wuliji

, & Yonemura

(2008a). Action! Update on the Global Pharmacy Education Consultation. International Pharmacy Journal , 22, 6–8. https://doi.org/10.1186/1478-4491-7-45.

Anderson

, Bates

, Beck

, Futter

, Mercer

, Rouse

, Wuliji

, & Yonemura

(2008b). The WHO UNESCO FIP Pharmacy Education Taskforce: enabling concerted and collective global action. American Journal of Pharmaceutical Education , 72(6), 127. https://doi.org/10.5688/aj7206127.

Anderson

, Bates

, Beck

, Manasse

H.R.

Jr. , Mercer

, Rouse

, Wuliji

, & Yonemura

(2006). FIP Roundtable Consultation on Pharmacy Education: Developing a Global Vision and Action Plan. International Pharmacy Journal , 20, 12–13.

Anderson

(2002). The state of the world’s pharmacy: a portrait of the pharmacy profession. Journal of Interprofessional Care , 16(4), 391–404. https://doi.org/10.1080/1356182021000008337.

Andrews

P.V.

, Schwarz

, & Helme

R.O.

(1992). Students can learn medicine with computers: Evaluation of an interactive computer learning package in geriatric medicine. Medical Journal of Australia , 157(10), 693–695.

10.

Angonesi

, & Renno

M.U.P.

(2011). Dispensação farmacêutica: proposta de um modelo para a prática. Ciência e Saúde Coletiva , 16(9): 3883–3891. https://doi.org/10.1590/S1413-81232011001000024.

11.

Audus

K.L.

, Moreton.

J.E.

, Normann

S.A.

, Sands

C.D.

, Seaba

H.H.

, Wincor

M.Z.

, Sagraves

, & Miller

K.W.

(2010). Going global: the report of the 2009–2010 research and graduate affairs committee 2010. American Journal of Pharmaceutical Education , 74(10), S8. https://doi.org/10.5688/aj7410s8.

12.

Awaisu

, & Mottram

D.R.

(2018). How Pharmacy Education Contributes to Patient and Pharmaceutical Care. In A.I. Fathelrahman, M.I.M. Ibrahim, A.A. Alrasheedy, & Wertheimer, A.I., Pharmacy Education in the Twenty First Century and Beyond: Global Achievements and Challenges (pp. 61–77). Cambridge, Academic Press.

13.

Baptista

, & Oliveira

(2018). Gamification and serious games: A literature meta-analysis and integrative model. Computers in Human Behavior , 92, 306–315. https://doi.org/10.1016/j.chb.2018.11.030.

14.

Barr

(2018). Student attitudes to games-based skills development: Learning from video games in higher education. Computers in Human Behavior , 80, 283–294. https://doi.org/10.1016/j.chb.2017.11.030.

15.

Berger

, Bawab

, de Mooij

, Sutter Widmer

, Szilas

, de Vriese

, & Bugnon

(2018). An open randomized controlled study comparing an online text-based scenario and a serious game by Belgian and Swiss pharmacy students. Currents in Pharmacy Teaching and Learning , 10(3), 267–276. https://doi.org/10.1016/j.cptl.2017.11.002.

16.

Bernardo

G.P.

, Bernardo

L.P.

, Barbosa

M.M.F.L

, Nascimento

T.J.

, Queiroz

N.L.P.

, Pezzi

L.H.A.

, Araúlo

, & Batista

H.M.T.

(2018). Simulatiom Based Education as Tool in Emergency Medicine: A Systematic Review. Amadeus International Multidisciplinary Journal , 3(5), 54–65. https://doi.org/10.14295/aimj.v3i5.50.

17.

Bigdeli

, & Kaufman

(2009). Digital games in medical education: Key terms, concepts, and definitions. Medical Journal of the Islamic Republic of Iran, 2017 , 31, 52. https://doi.org/10.14196/mjiri.31.52.

18.

Bindoff

, Ling

, Bereznicki

, Westbury

, Chalmers

, Peterson

, & Ollington

(2014). A computer simulation of community pharmacy practice for educational use. American Journal of Pharmaceutical Education , 78(9), 168. https://doi.org/10.5688/ajpe789168.

19.

Blakely

, Skirton

, Cooper

, Allum

, & Nelmes

Educational gaming in the health sciences: systematic review. Journal of Advanced Nursing , 65(2), 259–269. https://doi.org/10.1111/j.1365-2648.2008.04843.x.

20.

Boada

, Rodriguez-Benitez

, Thió-Henestrosa

, Olivet

, & Soler

(2018). How the gender of a victim character in a virtual scenario created to learn CPR protocol affects student nurses’ performance. Computer Methods and Programs in Biomedicine , 162, 233–241. https://doi.org/10.1016/j.cmpb.2018.05.019.

21.

Cain

, Conway

J.M.

, DiVall

M.V.

, Erstad

B.L.

, Lockman

P.R.

, Ressler

J.C.

, Schwartz

A.H.

, Stolte

, & Nemire

R.E.

(2014). Report of the 2013-2014 academic affairs committee. American Journal of Pharmaceutical Education , 78(10), S23. https://doi.org/10.5688/ajpe7810S23.

22.

Cant

R.B.

, & Cooper

S.J.

(2017). Use of simulation-based learning in undergraduate nurse education: An umbrella systematic review. Nurse Education Today , 49, 63–71. https://doi.org/10.1016/j.nedt.2016.11.015.

23.

Capdarest-Arest

(2019). “Game on!” Teaching gamification principles for library instruction to health sciences information professionals using interactive, low-tech activities and design-thinking modalities. Journal of the Medical Library Association , 107 (4), 566–571. https://doi.org/10.5195/jmla.2019.636.

24.

Carter

B.L.

(2016). Evolution of Clinical Pharmacy in the US and Future Directions for Patient Care. Drugs Aging , 33(3), 169–177. https://doi.org/10.1007/s40266-016-0349-2.

25.

Cochrane. Higgins

J.P.T

, & Green

(2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. Cochrane https://handbook-5-1.cochrane.org/.

26.

Cook

D.A

, Hamstra

S.J.

, Brydges

, Zendejas

, Szostek

J.H.

, Wang

A.T.

, Erwin

P.J.

, & Hatala

(2013). Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Medical Teacher , 35 (1), e867–98. https://doi.org/10.3109/0142159X.2012.714886.

27.

Crassd

R.L.

& Romanelli

(2018). Curricular reform in pharmacy education through the lens of the flexner report of 1910. American Journal of Pharmaceutical Education , 82(7), 6804. https://doi.org/10.5688/ajpe6804.

28.

Critical Appraisal Skills Programme. (2018). CASP Qualitative Research Checklist. http://media.wix.com/ugd/dded87_25658615020e427da194a325e7773d42.pdf.

29.

Croft

, Gilligan

, Rasiah

, Levett-Jones

, & Schneider

Current trends and opportunities for competency assessment in pharmacy education–a literature review. (2019). Pharmacy , 7(2), 67. https://doi.org/10.3390/pharmacy7020067.

30.

Cruz-Benito

, Maderuelo

, García-Peñalvo

F.J.

, Therón

, Pérez-Blanco

J.S.

, Zazo

, & Martín-Suárez

A..

(2016). A. Usalpharma: a software architecture to support learning in virtual worlds. Revista Iberoamericana de Tecnologias del Aprendizaje , 11(3), 194-204. https://doi.org/10.1109/RITA.2016.2589719.

31.

Curtin

L.B.

, Finn

L.A.

, Czosnowski

Q.A.

, Whitman

C.B.

, & Cawley

M.J..

(2011). Computer-based simulation training to improve learning outcomes in mannequin-based simulation exercises. American Journal of Pharmaceutical Education , 75 (6), 113. https://doi.org/10.5688/ajpe756113.

32.

Deshpande

A.A.

, & Huang

SH.

(2011). Simulation games in engineering education: A state-of-the-art review. Computer Applications in Engineering Education , 19(3), 399–410. https://doi.org/10.1002/cae.20323.

33.

Dominguez

, Saenz-de-Navarrete

, de Marcos

, Fernandez-Sanz

, Pages

, Martinez-Herraiz

J.J.

(2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education , 63, 380–92. https://doi.org/10.1016/j.compedu.2012.12.020.

34.

Dudzinski

, Greenhill

, Kayyali

, Nabhani

, Philip

, Caton

, Ishtiaq

, & Gatsinzi

(2013, October). The design and evaluation of a multiplayer serious game for pharmacy students. http://www.jesperjuul.net/text/gameplayerworld/.

35.

Durand

(1999). L’alchimie de La Compétence. Revue Française de Gestion , 127, 84–102. https://doi.org/10.3166/RFG.160.261-292.

36.

R.W.X.

, Yap.

K.Z.

, & Yap

K.Y.L.

(2018). Herbopolis–A mobile serious game to educate players on herbal medicines. Complementary Therapies in Medicine , 39, 68–79. https://doi.org/10.1016/j.ctim.2018.05.004.

37.

Ellis

L.B.

, & Hannigan

G.G.

(1986). Computers as learning resources in the health sciences: impact and issues. Bulletin of the Medical Library Association , 74(2), 95.

38.

Epstein

R.M.

, & Hundert

E.M.

(2002). Defining and assessing professional competence. Journal American Medical Association , 287(2), 226–235. https://doi.org/10.1001/jama.287.2.226.

39.

Fathelrahman

, Ibrahim

, & Wertheimer

(2016). Pharmacy practice in developing countries: Achievements and Challenges. London: Academic Press.

40.

Fonseca

L.M.

, Dias

D.M.

, Góes

F.D.

, Seixas

C.A.

, Scochi

C.G.

, Martins

J.C.

, & Rodrigues

M.A.

(2014). Development of the e-Baby serious game with regard to the evaluation of oxygenation in preterm babies: contributions of the emotional design. Computers, Informatics, Nursing , 32(9), 428–36. https://doi.org/ 10.1097/CIN.0000000000000078. Computers, Informatics, Nursing

41.

Fowler-Durham

& Alden

(Ed.). (2007). Enhancing patient safety in nursing education through patient simulation. In: Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville: Agency for Healthcare Research and Quality.

42.

Fox

, Pittaway

, & Uzuegbunam

(2018). Simulations in entrepreneurship education: Serious games and learning through play. Entrepreneurship Education and Pedagogy , 1(1), 61–89. https://doi.org/10.1177/2515127417737285.

43.

Frenk

, Chen

, Bhutta

Z.A

, Cohen

, Crisp

, Evans

, Fineberg

, Garcia

, Ke

, Kelley

, Kistnasamy

, Meleis

, Naylor

, Pablos-Mendez

, Reddy

, Scrimshaw

, Sepulveda

, Serwadda

, & Zurayk

(2010). Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. The Lancet , 376 (9756), 1923–58. https://doi.org/10.1016/S0140-6736(10)61854-5.

44.

Gaba

D.M.

(2004). The future vision of simulation in health care. Quality and Safety in Health Care , 13 (Suppl 1), i2–10. https://doi.org/10.1136/qshc.2004.009878.

45.

Gallagher

A.G.

, Ritter

E.M.

, & Satava

RM.

(2003). Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surgical Endoscopy and other Interventional Techniques , 17(10), 1525–1529. https://doi.org/10.1007/s00464-003-0035-4.

46.

Gallegos

, Tesar

A.J.

, Connor

, & Martz

K..

(2017). The use of a game-based learning platform to engage nursing students: A descriptive, qualitative study. Nurse Education in Practice , 27, 101–106. https://doi.org/10.1016/j.nepr.2017.08.019.

47.

Gentry

S.V.

, Gauthier

, Ehrstrom

B.L.E.

, Wortley

, Lilienthal

, Car

L.T.

, Dauwels-Okutsu

, Nikolaou

C.K.

, Zary

, Campbell

, & Car

(2019). Serious Gaming and Gamification Education in Health Professions: Systematic Review. Journal of Medical Internet Research , 21(3), e12994. https://doi.org/10.2196/12994.

48.

Gorbanev

, Agudelo-Londoño

, González

R.A.

, Cortes

, Pomares

, Delgadillo

, Yepes

F.J.

, & Muñoz

(2018). A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Medical Education Online , 23(1), 1438718. https://doi.org/10.1080/10872981.2018.1438718.

49.

Govaerts

M.J.

(2008). Educational competencies or education for professional competence? Medical Education , 42(3), 234–236. https://doi.org/10.1111/j.1365-2923.2007.03001.x.

50.

Graafland

, Schraagen

J.M.

, & Schijven

M.P.

(2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery , 99(10), 1322–1330. https://doi.org/10.1002/bjs.8819.

51.

Hawksworth

G.M.

, Corlett

A.J.

, Wright

D.J.

, & Chrystyn

(1999). Clinical pharmacy interventions by community pharmacists during the dispensing process. British Journal of Clinical Pharmacology , 47(6), 695. https://doi.org/10.1046/j.1365-2125.1999.00964.x.

52.

Hernández

, Garcia-Delgado

, Garcia-Cardenas

, Ocaña

, Labrador

, Orera

M.L.

, & Martinez-Martinez.

(2015). Characterization of patients’ requests and pharmacists’ professional practice in oropharyngeal condition in Spain. International Journal of Clinical Pharmacy , 37 (2), 300–9. . https://doi.org/10.1007/s11096-014-0053-4.

53.

Hill

L. H.

, Delafuente

J.C.

, Sicat

B.L.

, & Kirkwood

C.K.

(2006). Development of a competency-based assessment process for advanced pharmacy practice experiences. American Journal of Pharmaceutical Education , 70 (1), 01. https://doi.org/10.5688/aj700101.

54.

Holland

R.W.

, & Nimmo

C.M.

(1999). Transitions, part 1: beyond pharmaceutical care. American Journal of Health-System Pharmacy , 56(17), 1758–1764. https://doi.org/10.1093/ajhp/56.17.1758.

55.

Hookham

, Nesbitt

, Cooper

, Croft

, & Rasiah

(2015, February). Gamification for education: designing a pharmacy education game. https://link.springer.com/chapter/10.1007/978-3-319-14803-8_12.

56.

Hovancsek

(2007). Using simulation in nurse education. In Simulation in Nursing Education; from Conceptualization to Evaluation. New York: National League for Nursing.

57.

Ijaz

, Khan

M.Y.

, Ali

S.M.

, Qadir

, & Boulos

M.N.K.

(2019). Serious Games for Healthcare Professional Training: A Systematic Review. European Journal of Biomedical Informatics , 15(1). https://doi.org/10.24105/ejbi.2019.15.1.3.

58.

Inter-American Development Bank. Pedró

(2006). Aprender en el nuevo milenio: Un desafío a nuestra visión de las tecnologías y la enseñanza. Inter-American Development Bank https://publications.iadb.org/es/publicacion/13521/aprender-en-el-nuevo-milenio-un-desafio-nuestra-vision-de-las-tecnologias-y-la.

59.

International Pharmacy Federation. (2012). A global framework for services provided by Pharmacy workforce. Retrieved from International Pharmacy Federation. https://www.fip.org/files/fip/PharmacyEducation/GbCF_v1.pd.

60.

Issenberg

S.B.

& Scalese

R.J.

(2008). Simulation in health care education. Perspectives in Biology and Medicine , 51 (1), 31–46. https://doi.org/10.1353/pbm.2008.0004.

61.

Janarthanan

(2012, April). Serious video games: Games for education and health. https://ieeexplore.ieee.org/document/6209109.

62.

Juul

(2003, November). The game, the player, the world: Looking for a heart of gameness. http://www.jesperjuul.net/text/gameplayerworld/.

63.

Kim

, Sefcik

J.S.

, & Bradway

(2017). Characteristics of qualitative descriptive studies: a systematic review. Research in Nursing & Health , 40(1), 23–42. https://doi.org/10.1002/nur.21768.

64.

Kim

, Park

J.H.

, & Shin

(2016). Effectiveness of simulation-based nursing education depending on fidelity: a meta-analysis. BMC Medical Education , 16, 152. https://doi.org/10.1186/s12909-016-0672-7.

65.

Kinkade

R.E.

, Mathews

C.T

, Draugalis

J.R.

, & Erstad

B.L.

(1995). Evaluation of a Computer Simulation in a Therapeutics Case. American Journal of Pharmaceutical Education , 59, 147.

66.

Kleinert

, Wahba

, Chang

D.H.

, Plum

, Hölscher

A.H.

, & Stippel

D.L.

(2015). 3D immersive patient simulators and their impact on learning success: a thematic review. Journal of Medical Internet Research , 17(4), e91. https://doi.org/10.2196/jmir.3492.

67.

Kostenius

, Hallberg

, & Lindqvist

A.K.

(2018). Gamification of health education: Schoolchildren’s participation in the development of a serious game to promote health and learning. Health Education , 118(4), 354–368.

68.

Lam

J.T.

, Gutierrez

M.A.

, Goad

J.A.

, Odessky

, & Bock

(2019). Use of virtual games for interactive learning in a pharmacy curriculum. Currents in Pharmacy Teaching and Learning , 11(1), 51–57. https://doi.org/10.1016/j.cptl.2018.09.012.

69.

Lambertsen

R.T.

, Tang

, Davies

, & Morecroft

(2016). Serious gaming for pharmacy education: Development of a serious games for teaching pharmacist communication and drug administration in a virtual hospital setting. 2016 9th International Conference on Developments in eSystems Engineering, 151–156. http://dx.doi.org/10.1109/DeSE.2016.25.

70.

Lee

C.Y.

, White

P.J.

, & Malone

D.T.

(2018). Online educational games improve the learning of cardiac pharmacology in undergraduate pharmacy teaching. Pharmacy Education , 18, 298–302.

71.

Liao

Y. W.

, Huang

Y. M.

, & Wang

Y. S.

(2015). Factors affecting students’ continued usage intention toward business simulation games an empirical study. Journal of Educational Computing Research , 53(2), 260–283.

72.

Liao

Y.W.

, & Wang

Y.S.

(2011). Investigating the factors affecting students’ continuance intention to use business simulation games in the context of digital learning. International Conference on Innovation, Management and Service, 119–24.

73.

Lucchese

, & Ribeiro

(2009). Conceituação de jogos digitais. http://www.dca.fee.unicamp.br/~martino/disciplinas/ia369/trabalhos/t1g3.pdf.

74.

Malhotra

V.M.

, Kabra

P.R.

, & Malhotra

(2017). Attitudes and practices of medical students regarding video-games: should community medicine educationists get serious about serious games. International Journal Of Community Medicine And Public Health , 4(3), 729–733. https://doi.org/10.18203/2394-6040.ijcmph20170748.

75.

Martins

S.F.

, van Mil

J.W.F.

, & Costa

F.A.

(2015). The organizational framework of community pharmacies in Europe. International Journal of Clinical Pharmacy , 37(5), 896–905. https://doi.org/10.1007/s11096-015-0140-1.

76.

Martí-Parreño

, Galbis-Córdova

, & Miquel-Romero

M.J.

(2018). Students’ attitude towards the use of educational video games to develop competencies. Computers in Human Behavior , 81, 366–377. https://doi.org/10.1016/j.chb.2017.12.017

77.

Mayer

, Warmelink

, & Bekebrede

(2013). Learning in a game-based virtual environment: A comparative evaluation in higher education. European Journal of Engineering Education , 38(1), 85–106. https://doi.org/10.1080/03043797.2012.742872.

78.

Mesquita

A.R.

, Souza

W.M.

, Boaventura

T.C.

, Barros

I.M.

, Antoniolli

A.R.

, Silva

W.B.

, & Júnior

D.P.L.

(2015). The effect of active learning methodologies on the teaching of pharmaceutical care in a Brazilian pharmacy faculty. PLoS One , 10(5), e0123141. https://doi.org/10.1371/journal.pone.0123141.

79.

Miller

C.T.

(Ed.). (2008). Games: Purpose and potential in education (1st ed.). Ney York: Springer Science & Business Media.

80.

Mills

J.A.

, Middleton

J.W.

, Schafer

, Fitzpatrick

, Short

& Cieza

(2020). Proposing a re-conceptualisation of competency framework terminology for health: a scoping review. Human Resources for Health , 18, 15. https://doi.org/10.1186/s12960-019-0443-8.

81.

Minhas

P.S.

, Ghosh

, & Swanzy

(2012). The effects of passive and active learning on student preference and performance in an undergraduate basic science course. Anatomical Sciences Education, (4), 200–7. https://doi.org/10.1002/ase.1274.

82.

Ministry of Health of Brazil. National Drug Policy (Politíca Nacional de Medicamentos), Portaria 3.916, 30 October 1998, Brasilia; 1998. http://bvsms.saude.gov.br/bvs/saudelegis/gm/1998/prt3916_30_10_1998.html.

83.

Moher

, Liberati

, Tetzlaff

, & Altman

D.G.

(2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine , 151(4), 264–269. https://doi.org/10.1371/journal.pmed.1000097.

84.

Nara

Beppu

, Tohda

, & Suzuki

(2009). The introduction and effectiveness of simulation-based learning in medical education. Internal Medicine, 48 (17),1515–9. https://doi.org/10.2169/internalmedicine.48.2373.

85.

National Center For Biotechnology Information website. MeSH – Medical Subject Headings (2018). NCBI http://www.ncbi.nlm.nih.gov/mesh.

86.

National Health Service. NHS Community Pharmacy Services – a summary. NHS http://psnc.org.uk/contract-it/the-pharmacy-contract/.

87.

Niemi

(2002). Active learning—a cultural change needed in teacher education and schools. Teaching and Teacher Education , 18(7), 763–780. https://doi.org/10.1016/S0742-051X(02)00042-2.

88.

Noori

, Kouti

, Akbari

, Assarian

, Rakhshan

, & Eslami

(2014). A review on different virtual learning methods in pharmacy education. Journal of Pharmeutical Care , 2 (2), 77–82.

89.

Ouzzani

, Hammady

, Fedorowicz

, & Elmagarmid

(2016). Rayyan—a web and mobile app for systematic reviews. Systematic Reviews , 5(1), 210. https://doi.org/10.1186/s13643-016-0384-4.

90.

Payne

M.B.

, & Rowberg

A.H.

(1984). A Computer Learning Game for Nurses. Proceedings of the Annual Symposium on Computer Application in Medical Care, 953–955.

91.

Ravyse

W.S.

, Blignaut

A.S.

, Leendertz

, & Woolner

(2017). Success factors for serious games to enhance learning: a systematic review. Virtual Reality , 21(1), 31–58. https://doi.org/10.1007/s10055-016-0298-4.

92.

Reed

D.A.

, Cook

D.A.

, Beckman

T.J.

, Levine

R.B.

, Kern

D.E.

, & Wright

S.M.

(2007). Association between funding and quality of published medical education research. Journal American Medical Association , 298, 1002–1009.

93.

Rienzo

& Cubillos

(2020). Playability and Player Experience in Digital Games for Elderly: A Systematic Literature Review. Sensors , 20 (14), 3958. https://doi.org/10.3390/s20143958.

94.

Rodriguez

D.M.

, Teesson

, & Newton

N.C.

(2014). A systematic review of computerised serious educational games about alcohol and other drugs for adolescents. Drug and Alcohol Review , 33(2), 129–35. https://doi.org/10.1111/dar.12102.

95.

Sangrà

, & González-Sanmamed

(2010). The role of information and communication technologies in improving teaching and learning processes in primary and secondary schools. ALT-J , 18(3), 207–220. https://doi.org/10.1080/09687769.2010.529108.

96.

Sardi

, Idri

, & Fernández-Alemán

J.L.

(2017). A systematic review of gamification in e-Health. Journal of Biomedical Informatics , 71, 31–48. https://doi.org/10.1016/j.jbi.2017.05.011.

97.

Saseen

J.J.

, Ripley

T.L.

, Bondi

, Burke

J.M.

, Cohen

L.J.

, McBane

, McConnell

K.J.

, Sackey

, Sanoski

, Simonyan

, Taylor

, & Griend

J.P.V.

(2017). ACCP Clinical Pharmacist Competencies. Pharmacotherapy , 37(5), 630–636. https://doi.org/10.1002/phar.1923.

98.

Schijven

M.P.

, & Jakimowicz

J,J.

(2005). Validation of virtual reality simulators: Key to the successful integration of a novel teaching technology into minimal access surgery. Minimally Invasive Therapy & Allied Technologies , 14(4-5), 244–246. https://doi.org/10.1080/13645700500221881.

99.

Sera

, & Wheeler

(2017). Game on: The gamification of the pharmacy classroom. Currents in Pharmacy Teaching and Learning , 9(1): 155–159. https://doi.org/10.1016/j.cptl.2016.08.046.

100.

Serrano-Laguna

Á.

, Martínez-Ortiz

, Haag

, Regan

, Johnson

, & Fernández-Manjón

(2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces , 50, 116–123. https://doi.org/10.1016/j.csi.2016.09.014.

101.

Shea

B.J.

, Reeves

B.C.

, Wells

, Thuku

, Hamel

, Moran

, Moher

, Tugwell

, Welch

, Kristjansson

, & Henry

D. A.

(2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal , 358, j4008. https://doi.org/10.1136/bmj.j4008.

102.

Sipiyaruk

, Gallagher

J.E.

, Hatzipanagos

, & Reynolds

P.A.

(2018). A rapid review of serious games: From healthcare education to dental education. European Journal of Dental Education , 22(4), 243–257. https://doi.org/10.1111/eje.12338.

103.

Smith

C.E.R.

, Ryder

, Bilodeau

, & Schultz

(2016). Use of an online game to evaluate health professions students’ attitudes toward people in poverty. American Journal of Pharmaceutical Education , 80(8), 139. https://doi.org/10.5688/ajpe808139.

104.

Smith

C.E.R.

,., Ryder

, Blais

, & Schneck

(2017). Evaluation of two different poverty simulations with professional phase pharmacy students. Currents in Pharmacy Teaching and Learning, 2017; 9(5), 903–910. https://doi.org/10.1016/j.cptl.2017.05.018.

105.

Sullivan

G.M.

(2011). Deconstructing quality in education research. Journal of Graduate Medical Education , 3(2), 121–124. https://doi.org/10.4300/JGME-D-11-00083.1.

106.

Tandogan

R.O.

, & Orhan

(2007). The Effects of Problem-Based Active Learning in Science Education on Students’ Academic Achievement, Attitude and Concept Learning. Journal of Mathematics, Science & Technology Education , 3(1), 71–81. https://doi.org/10.12973/ejmste/75375.

107.

Toklu

H.Z.

, & Hussain

(2013). The changing face of pharmacy practice and the need for a new model of pharmacy education. Journal of Young Pharmacists , 5(2), 38–40. https://doi.org/10.1016/j.jyp.2012.09.001.

108.

Tomé Filho

F.R.

, Kapralos

, Pejman

M.B.

, Mendonça Júnior

G.M.

(2019). Let’s Play Together: Adaptation Guidelines of Board Games for Players with Visual Impairment. CHI Conference on Human Factors in Computing Systems Proceedings, 631. https://doi.org/10.1145/3290605.3300861.

109.

Uskoković

(2017). Rethinking active learning as a paradigm of our times: towards poeticizing and humanizing natural sciences in the age of stem. Journal of Materials Education , 39 (5-6), 241–258.

110.

van Mil

J.F.

, Schulz

, & Tromp

T.F.D.

(2005). Pharmaceutical care, European developments in concepts, implementation, teaching, and research: a review. Pharmacy World and Science , 26(6), 303–311. https://doi.org/10.1007/s11096-004-2849-0.

111.

Verkuyl

, Atack

, Mastrilli

, & Romaniuk

(2016). Virtual gaming to develop students’ pediatric nursing skills: A usability test. Nurse Education Today , 46, 81-85. https://doi.org/10.1016/j.nedt.2016.08.024.

112.

Vilela

V.V.

, Barbosa

L.C.P.

, Miranda-Vilela

A.L.

, & Santos Neto

L.L.

(2013). The use of mind maps as support in medical education. Journal of Contemporary Medical Education , 1(4), 199–206. https://doi.org/10.5455/jcme.20130407082647.

113.

Vlachopoulos

& Makri

(2017). The effect of games and simulations on higher education: a systematic literature review. International Journal of Educational Technology in Higher Education , 14 (1), 22. https://doi.org/10.1186/s41239-017-0062-1.

114.

Vyas

, Bray

B.S.

, & Wilson.

N.M.

(2013). Use of simulation-based teaching methodologies in US colleges and schools of pharmacy. American Journal of Pharmaceutical Education , 77 (3), 53. https://doi.org/10.5688/ajpe77353.

115.

Wang

, DeMaria

Jr , Goldberg

, & Katz

(2016). A systematic review of serious games in training health care professionals. Simulation in Healthcare , 11(1), 41–51. https://doi.org/10.1097/SIH.0000000000000118.

116.

Warren

J.N.

, Luctkar-Flude

, Godfrey

, & Lukewich

(2016). A systematic review of the effectiveness of simulation-based education on satisfaction and learning outcomes in nurse practitioner programs. Nurse Education Today , 46, 99–108. https://doi.org/10.1016/j.nedt.2016.08.023.