Abstract
Introduction. As
Keywords
Introduction
GridlockED is a serious game (Crookall, 2010; Graafland et al., 2012; Laamarti et al., 2014; Olszewski & Wolbrink, 2017), designed to emulate an 8-hour shift in the emergency department (ED) (Tsoy et al., 2019). Since the spring of 2018 this game has been available for purchase and has been distributed around the world. In the game, players collaboratively work to manage the care of a variety of patients presenting with differing levels of acuity and severity of illness, represented in the game by patient cards. The intended target audience was initially health professions educators, as we had designed the game as a platform for educators to teach two main concepts: 1) how patients flow (or do not flow) through the ED and 2) how an interprofessional team of providers work together to care for patients
In GridlockED, players are in control of the various healthcare practitioners who work in the ED (a physician, a resident, a consultant, a radiologist, and four nurses). They move these characters around the board in order to carry out actions necessary for managing patient care. The goal is two-fold: 1) survive the shift by keeping patients safe and avoiding patient safety events and 2) admit or discharge patients to accrue 500 points for your ED. Each turn represents an hour of time in the ED, and each character is allotted a certain number of actions per turn (representing labour capacity per hour). Additionally, event cards are shuffled in alongside patient cards that are randomly drawn from a face-down deck, and players must respond to these events, some of which are good (e.g. additional staffing is available) and while others are bad luck (e.g. a neighboring hospital redirects extra ambulances to you).
In this study, our goal was to better understand the demographics and game usage by potential and actual purchasers of GridlockED, as well as identify key barriers to purchasing or using the game for its educational intent. There are few reports of how serious games in healthcare education are used by end-users beyond the original testing and evaluation of the game (Graafland et al., 2012; Guillén-Nieto & Aleson-Carbonell, 2012; Jarvis & de Freitas, 2009; Olszewski & Wolbrink, 2017). Post-market surveillance is seldom conducted to determine how and with whom end-users are using their educational products. As with all products, how the designers intend to use the product and how end-users may use the product can be vastly different. Once in the hands of users around the world, we were unsure of whether they would use GridlockED with our specific intent or if they would use it to teach or learn different concepts. For instance, are there individuals who purchase this for intentions other than for education? Our goal was to use this study to inform the continued improvement of our present game, and to better understand our end-users for further game development, carving out a new type of entrepreneurial scholarship that might give insights to other game developers.
Methods
Study Design
We designed a two-phase survey-based quality assurance study to capture information about potential and actual purchasers of the GridlockED game. These surveys were designed to be uniquely suited to our product and its usage cases. The surveys were initially pilot tested with educational experts from several key stakeholder groups (academic physicians, program directors), and feedback from these individuals resulted in amendments and improvements to the survey.
Phase 1
Upon arrival to the game’s website (www.gridlockedgame.com) those interested in purchasing the game were asked to answer an intake survey (Survey 1) which registered them into our potential purchaser database. All participants were asked to answer a few key questions about their present role, their intent for purchasing the game, and their location (city, state/province, country). After answering these three questions, participants were asked if they would like to participate in a quality assurance follow-up survey, providing those not interested in participating in a follow-up questionnaire with the option to opt-out. See appendix 1 for details of the intake survey.
Phase 2
Those individuals who consented to the follow-up questionnaire were emailed between 3-6 months after completing the intake survey. To maximally capture those who had completed the intake form and purchased the game, we launched this survey approximately 3 months after the first responses on the intake form, and roughly every 3 months thereafter, providing reminders monthly (Dillman, 2007). This follow-up survey was sent from the GridlockED game email account, and contained questions that asked for demographics (e.g. their age, role, and location) and how the respondents had used the game to date. Those who did not purchase the game were asked about the barriers to purchasing, while those who did purchase the game were asked about their game usage, with whom they had played, and what they liked or not about the game, or future directions. See appendix 2 for details of the follow-up survey.
Analysis
We conducted simple descriptive statistics of our survey results using Microsoft Excel (Microsoft Corp., Redmond, WA, USA). In addition, IBM SPSS version 25.0 (IBM Corp., Armonk, NY, USA) was used to explore correlations between roles reported in the intake survey and anticipated usages of the game (e.g. Pearson correlations). We also conducted various comparisons between academic/educators’ and community practitioners’ usage.
For the free text entries, a simple thematic analysis was conducted by a single author (TMC) to determine key themes noted with the two free-text questions. Initially, she combined all of the data into a single repository and began theming the data into groups and inductively creating headings and themes that described groups of comments. To ensure the rigor of the analysis, an audit of their notes by another author (JBB) was conducted. Discrepancies were resolved via a consensus building process.
Ethics
We received a quality assurance exemption for this study from our institutional review board (Hamilton Integrated Research Ethics Board).
Results
Phase 1 Results: The Intake Survey
Over the course of our first year of sales (March 27, 2018-March 26, 2019), a total of 560 unique respondents completed the intake survey, and of these individuals 213 purchased at least one copy of the game (37.2%). Of those individuals who completed the intake survey, 408 (72.6%) consented to being contacted at a later point. The remaining individuals (152, 28.4%) chose to opt out of the follow-up survey. Table 1 shows the demographics of those who displayed initial interest in purchasing the game. The groups showing the most amount of interest in potentially purchasing the game were: Academic EM physicians/consultants (n = 92, 16.4%), EM trainees (n = 58, 10.4%), and Community EM physicians (n = 58, 10.4%).
Roles of Individuals Expressing Interest in Purchasing the GridlockED Board Game.
Legend: EM = Emergency Physician, PD = Program Director, ED = Emergency Department.
Total is greater than 100% as some individuals reported multiple roles.
NB: Although there are smaller percentages for some professions, we have included all participants who responded to our intake survey. We hope that providing an overview of all participants will further future potential studies on the diverse audience that may be interested in serious healthcare board games.
The majority of respondents were located in the USA (n = 185, 33%), Canada (n = 179, 32%), the UK (n = 35, 6.3%), and Australia (n = 33, 5.9%). Figure 1 shows a map of all the countries where individuals completed the intake survey.

A heat map of the GridlockED intake survey respondents. The level of shades of blue represent number of inquiry responses.
The most popular reasons respondents stated for wanting to purchase the game was to teach trainees (49.6%, n = 278); to conduct simulations to consider process or quality improvement (36.4%, n = 204); to teach colleagues (n = 28.9%, n = 162). Just over a third of individuals reported intending to use the game for fun rather than explicit training or educational purposes (36.8%, n = 206). Less commonly, individuals foresaw using the game to explain their jobs to their family and friends (13.9%, n = 78) or giving it as a gift (12.9%, n = 72). Table 2 summarizes the intended usages reported by respondents who completed the intake form.
Intended Usage Anticipated by Respondents in GridlockED Intake Survey.
NB: These percentages total more than 100% as respondents could report multiple intended usages.
Correlation analysis
There was only a weak, but significant, Pearson correlation between the number of copies purchased and the intention to teach trainees (r = 0.14, p < 0.01) or pre-health and secondary school students (r = 0.12, p < 0.01). There were no other clear associations between the other intended usages and actual copies purchased.
In terms of their intended usages, 46% (n = 256) respondents in the intake survey reported anticipating more than one use for GridlockED. Generally speaking, those who indicated multiple purposes envisioned using the game for teaching trainees (Pearson’s r = 0.35, p < 0.01), teaching colleagues (r = 0.56, p < 0.01), simulating the ED for process improvement exercises (r = 0.53, p < 0.01), explaining their job to their friends and family (r = 0.42, p < 0.01), and playing the game for fun (r = 0.41, p < 0.01).
There was a moderate correlation between either those who intended to teach colleagues and those who intended to teach trainees (r = 0.37, p < 0.01) or use GridlockED as a platform to engage in simulation for process improvement (r = 0.34, p < 0.01). There was also a moderate association between those who were interested in playing the game for fun and using it to explain their job to their family members (r = 0.36, p < 0.01). Adjustments to the threshold for significance have been made to account for the multiple comparisons per the Bonferroni method (Norman & Streiner, 2013).
There was a weak negative correlation (r = –0.16, p < 0.01) between using the game for fun and harnessing it as a platform to teach trainees. Similarly, the intention to give the game as a gift was negatively correlated with the intention to use it for teaching trainees (–0.20, p < 0.001) and for process improvement simulations (r = –0.14, p < 0.01). There was a weak correlation (r = 0.12, p < 0.01) between using the game as a teaching tool for trainees and dual purposing it as a simulation tool for process improvement.
Comparisons between groups
We found that there was a difference between the purchasing patterns of those who identified as educators/academics or those who identified as community physicians or non-educator practitioners (Pearson Chi-Squared df(1) = 38.9, p < 0.001 when comparing fun purchases, and df(1) = 26.5, p < 0.001 when comparing teaching trainee intentions). Academics and educators purchased the game more frequently for teaching purposes (70% of the time) and less frequently for fun (20.7%), whereas community physicians and non-educator healthcare practitioners (e.g. nurses, paramedics, etc.) purchased the game less frequently for teaching (39.8%) and more frequently for fun (55.3%).
Phase 2 Results: The Follow-Up Survey
From June 15, 2018-July 21, 2019, a total of 53 respondents (13.0% of the 408 who consented to being contacted in follow-up) participated in the follow-up survey (Survey 2). Out of these follow-up survey respondents, a total of 33 participants had purchased and received a copy of GridlockED whereas the other 20 had either opted out of purchasing or had not yet received the game.
Respondents to the follow-up survey were from 12 different countries. The majority of respondents were from Canada (38%), the USA (25%), New Zealand (9%), and Turkey (6%). The breakdown of the countries of origin for our follow-up survey respondents is shown in Table 3.
Themes From Free-Text Analysis About Respondents Perceptions About the Best and Most Problematic Aspects of GridlockED.
Legend: ED = Emergency Department; dept = Department.
Of those who purchased the game, only 37.7% (n = 20/53) had played it as intended (e.g. as a teaching tool), suggesting that several individuals (24.6%, n = 13/53) had used the game in ways beyond our initial intention. Within the group that used it as a teaching tool, 20.8% (n = 11/53) reported that they had used it to teach students and trainees, 7.5% (n = 4/53) had used it to teach colleagues, and 5.7% (n = 3/53) had used it as a simulation for quality or process improvement. A substantial subgroup (39.6%, n = 21/53) of those who had played also noted that they had played the game for fun. Participants reported playing with their spouse (n = 3/53) or friends (n = 4/53).
Table 3 shows our thematic analysis of the comments found in the free-text portions of the follow-up survey. Five individuals simply noted that they loved the concept of the game. Notably, many respondents (n = 14) remarked on how well the game simulated the real emergency department. A number of individuals (n = 6) also reported that the best aspect of GridlockED was its ability to teach flow concepts and managerial/leadership skills.
For those who did not purchase the game (n = 20), price of the game unit itself was the largest barrier (60%, n = 12/20), followed by the cost of shipping for the game unit (40%, n = 8/20). The majority of these individuals (55%, n = 11/20) stated that if the game were cheaper, they would consider buying it in the future.
Figure 2 shows a heat map of the follow-up survey responses. Of the game purchasers who responded to the follow-up survey (n = 33), the vast majority of the group was interested in a future disaster preparedness expansion pack for the current game (85.3%, n = 29/33) and a number of individuals were interested in seeing development of a teacher’s guide (i.e. lesson plans) for the current game (63.6%, n = 21/33).

A heat map of the GridlockED follow-up survey respondents. The level of shades of red represent number of survey responses.
Discussion
The design and sales of GridlockED are a window into a new type of entrepreneurial scholarship that may be of benefit for other serious game producers to consider. The price of the game and shipping costs may be problematic for the potential audience of educators. Since purchasers had often used the game for our intended purposes to teach trainees (37.7%, n = 20/53), the cost of a game may be a barrier that limits the accessibility of educational tools such as serious games for many educators. Creating a version of the game that is aligned with the Free Open Access Medical education (FOAM) (Cadogan et al., 2014; Nickson & Cadogan, 2014) movement may be of benefit, and potentially broaden the reach of serious games like GridlockED.
Negative correlation in the intake form data between those who intended to purchase the game for teaching purposes and those who intended to use the game for fun suggests that there may be two unique markets: the intended audience (e.g. teachers, trainers, healthcare leaders) and an unintended audience (e.g. those buying the game for fun, family purchasing the item as a gift). This may explain the number of comments requesting more detailed instructions – what may be obvious to our intended healthcare audience may not be apparent to the unintended audience. Similarly, our chi-squared comparisons between groups nicely triangulated the case that these groups are likely divergently different. Serious games should consider taking this potential secondary, divergent audience into account as part of the design phase. Previously literature has suggested that there is potential for substantive overlap and blurring of intents for sports-based or board games to be used for fun or for more serious intents (Laamarti et al., 2014).
Finally, even though many users may initially intend to purchase the serious game for teaching or clinical purposes, after purchasing they may still play the game for fun. We found that a high proportion of those who purchased the game (36.9%, n = 21/53) had used it for its top off-label purpose: Fun!
Our findings suggest that although our intentions were to make a serious game, many individuals may be using it for entertainment purposes. Therefore, when designing a serious game it may be useful to consider ensuring that the game has some challenging and random elements that might entice players to play the game more than once (Dunwell et al., 2011; Kelly et al., 2007). We suspect that although participants are reporting that they are playing for fun, the learning aspects of the serious game may spill over into these ‘fun’ rounds of play.
A deeper dive into the patterns within the differences in the various groups’ (e.g. academic vs. community practitioners) may reveal better explanation as to why certain individuals cited that our price point was deemed a barrier to sales. For instance, with our present de-linked data, we cannot pair our follow-up to our initial survey, but we anticipate that those purchasing the game for amusement may find that the present price is too expensive, whereas an educator who is comparing our price point to a simulation manikin may find our product quite reasonably priced. Future work in this area may help to clarify how and what we might do to target or adjust our products for the different consumer groups.
Finally, our present study revealed to us that end-users of serious games may have multiple intentions when purchasing the item. As with most endeavours, it is imperative for our team to learn from our experiences, and we hope that our findings might be of interest to our field at large. Although serious games for healthcare have existed for quite some time (Graafland et al., 2012; Jarvis & de Freitas, 2009; Knight et al., 2010; Laamarti et al., 2014), there are few scholarly reports in the literature regarding end-user intentions or purchasing patterns (Ho & Wu, 2012). Our study reveals some potentially important patterns in the intentions of potential game purchasers that may be useful to other serious game designers hoping to take their games to market.
Limitations
Our study has a number of limitations. First off, we have only presented our first year of data. As we have had many purchases after this first year, the first round of surveys may not be completely generalizable to subsequent data. Also, our intake-survey is skewed towards those who are already interested in the game, since those who visited the website were the only individuals who would access the intake survey. This group may be different than a more general population of healthcare educators, since they may be more interested in gaming or be more internet-savvy. Our follow-survey had a small number of respondents despite multiple attempts to recruit their engagement. The poor response rate will limit the generalizability of our findings, since it is likely not representative writ large of the greater GridlockED purchaser pool, but perhaps will serve as a first step for research in this domain.
In addition, since GridlockED was based on emergency care in the Canadian context, it applies a very North American perspective on how emergency care is provided. Thus, even though we have had purchases all across the globe, the applicability of the game for all the different environments may have limited the response rate for our follow-up survey. This bias might have skewed our results in favour of the more serious use cases (e.g. teaching, simulations for improving clinical care) rather than more trivial uses (e.g. fun, explaining your job to your family). To minimize this bias, we allowed participants to choose more than one intention in both the intake and follow-up forms.
Conclusion
The analysis of the pre- and post-purchase (or non-purchasing) populations of our serious game (GridlockED) has provided us with some insights into the duality of the market for such games. Based on our findings, we highlight the two populations that exist: those who purchase the game for amusement, and those who purchase the game with more academic intentions. Bearing these populations in mind, designers and those interested in marketing their games may want to attend to these two overlapping populations.
Supplemental Material
GridlockED_Game_-_Follow-up_Survey_-_Google_Forms – Supplemental material for Side Effects May Include Fun: Pre- and Post-Market Surveillance of the GridlockED Serious Game
Supplemental material, GridlockED_Game_-_Follow-up_Survey_-_Google_Forms for Side Effects May Include Fun: Pre- and Post-Market Surveillance of the GridlockED Serious Game by Stephen J. Hale, Sonja Wakeling, J. Bruce Blain, Alim Pardhan, Shawn Mondoux and Teresa M. Chan in Simulation & Gaming
Supplemental Material
GridlockED_Game_Purchase_Portal_-_Google_Forms_Redacted – Supplemental material for Side Effects May Include Fun: Pre- and Post-Market Surveillance of the GridlockED Serious Game
Supplemental material, GridlockED_Game_Purchase_Portal_-_Google_Forms_Redacted for Side Effects May Include Fun: Pre- and Post-Market Surveillance of the GridlockED Serious Game by Stephen J. Hale, Sonja Wakeling, J. Bruce Blain, Alim Pardhan, Shawn Mondoux and Teresa M. Chan in Simulation & Gaming
Footnotes
Author Contribution
SJH, TMC, AP made substantial contributions to conception and design. SJH, TC, SM contributed to the acquisition of data, while SJH, JBB, AP, TMC, SW contributed to the analysis and interpretation of data. All authors drafted the article or revised it critically for important intellectual content. All authors will have approval the final version to be published.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Author Biographies
Contact:
Contact:
Contact:
Contact:
Contact:
Contact:
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
