Causal factors of low stakeholder engagement: a survey of expert opinions in the context of healthcare simulation projects

Abstract

While simulation methods have proved to be very effective in identifying efficiency gains, low stakeholder engagement creates a significant limitation on the achievement of simulation modeling projects in practice. This study reports causal factors—at two hierarchical levels (i.e., primary and secondary)—that could significantly affect low stakeholder engagement in healthcare simulation projects. A self-completed questionnaire was administered online to 91 experts in the field from whom 37 responded. The results were reinforced using a bootstrapped sample (n = 1000). Data were analyzed using descriptive statistics, Kendal’s tau-b correlations, and non-linear multiple regression. Based on our research, while such factors as “communication gap”, “stakeholders’ high workload”, and “too much complexity involved” represent the most significant primary causal factors, some others such as “reluctance to change” proved interestingly insignificant. The research suggests that high-impact public health projects can exemplify the areas that will potentially generate engagement in the healthcare simulation domain.

Keywords

Stakeholder engagement causal factors simulation project healthcare survey

1. Introduction

In many countries, there appears to be substantial scope to improve the health status of the population without increasing spending, but instead through efficiency gains.¹ The empirical results of an OECD (Organisation for Economic Co-operation and Development) study suggest that potential efficiency gains might be large enough to raise life expectancy at birth by almost three years on average for OECD countries, while a 10% increase in total health spending would increase life expectancy by only three to four months.¹

Simulation methods have proved to be very effective in identifying efficiency gains. Computer simulation has been arguably the second most widely used research method in the field of operations management.² Simulation methods have historically played a major role in system improvement initiatives in non-healthcare settings. The situation in healthcare, however, is mixed. While there has been much activity there particularly on system utilization, policy decision support and public health,³ limited evidence of implementation exists.^3–6 An example to demonstrate the potential, however, is presented by Roberts et al.,⁷ where it is reported that by simulating changes to the current care pathway of end-stage renal disease, shifts from medical center dialysis to either home dialysis or cadaver donor transplantation would save US$284 million per year. The literature demonstrates the application of different simulation methods—mostly Monte Carlo simulation, discrete event simulation, system dynamics (SD), and agent-based simulation—in healthcare.⁸

A project success is about bringing into play the interests of those who established the project and realizing their expectation of project achievement.⁹ Failure to meet stakeholders’ expectations in the simulation projects has been a common phenomenon, mainly as a result of lack of shared views between simulation providers and customers about the concept of success in the simulation projects.¹⁰ The literature provides strong evidence that “user/stakeholder engagement and support” is the critical factor in the successful ending of various change projects in general,^11–20 and simulation projects in particular.^21,22

Recent studies in the context of healthcare suggest that “low stakeholder engagement” has created a significant limitation on the achievement of simulation modeling projects.^22,23 Yet, our search—as explained in the next section—failed to identify any studies dedicated to the investigation of possible causal factors for this. The present article therefore aims to empirically ascertain the most important causes of low stakeholder engagement in healthcare simulation projects through a survey of expert opinions. Such an investigation will result in a better understanding of the issue and then a more informed approach to tackle it.

We begin in Section 2 with a summary of research background and its underlying theoretical framework. In Section 3, we outline our methods that are used to conduct the research. Section 4 presents the survey results and analysis. Section 5 presents a discussion of findings and suggestions on the implications for research and practice. Section 6 brings the paper to a close and suggests future work that might be carried out.

2. Background and theoretical framework

What we mean by stakeholders in this context, as inspired by Freeman’s definition,²⁴ is “individuals, groups or organisations that directly or indirectly can affect or be affected by (positively or negatively) the simulation modelling project”. More information on the definitions, with a particular focus on the healthcare contexts, can be seen in Brailsford et al.²²

Research evidence on the stakeholder engagement within the healthcare simulation domain is very limited, and mainly in the form of scattered information throughout the literature on the challenges of simulation modeling in general^{5,18,21,25–27} or in some specific contexts such as healthcare.^{22,23,28–33} However, there is a clear gap on the “causal study” of “low stakeholder engagement”. The present study attempts to fill this gap in the body of knowledge, which would take the research one step further towards solving the issue.

From the published literature³⁴ and using the authors’ experience in the subject matter, we identified 18 primary and 44 secondary factors that are expected to have an effect on stakeholder engagement in healthcare simulation projects. These factors establish our research hypotheses that need to be tested in the research. Table 1 presents these factors along with the literature sources. We primarily grouped these factors into four categories (i.e., organizational, technical, project management, and healthcare-specific factors) based on agreement between two co-authors, each having more than 10 years of experience in simulation modeling. Organizational category principally represents the indigenous factors inherent in the host organization that is the recipient of simulation work. The technical factors represent the issues specifically concerned with the simulation tools and techniques. The project management category is mainly associated with exogenous factors originating from the simulation provider institution and its project management style. The healthcare-specific category, however, is different in a way that it has a focus on the factors that are mostly associated with the healthcare settings.

Table 1.

Hypothetical primary and secondary casual factors affecting stakeholder engagement in healthcare simulation projects

	Code	Primary causal factors (PC)	Sources	Code	Secondary causal factors (SC)^a
Organizational	PC1	Reluctance to change	[28]	SC1.1	Strict organizational rules and codes of practice do not support change and innovations
	PC1	Reluctance to change	[28]	SC1.2	Fear of instability
	PC2	Stakeholder’s (e.g., clinician’s) high workload	[22, 32]	SC2.1	Shortage of workforce
	PC2	Stakeholder’s (e.g., clinician’s) high workload	[22, 32]	SC2.2	Lack of proper workforce planning
	PC3	Poor familiarity or awareness of simulation	[26]	SC3.1	Low exposure to simulation projects
	PC3	Poor familiarity or awareness of simulation	[26]	SC3.2	Lack of background knowledge about simulation
	PC4	Conflicting views and objectives amongst healthcare community when discussing managerial issues	[18, 29, 32]	SC4.1	Difficulty in aligning views and objectives
	PC4		[18, 29, 32]	SC4.2	Poor communication amongst stakeholders
	PC5	Communication gap between simulation and stakeholder groups	[5, 10, 10]	SC5.1	Technical language barrier
				SC5.2	Poor interactive tools or modes of communication
				SC5.3	Irregular contacts between two groups
				SC5.4	Poor awareness of other group’s roles
	PC6	Stakeholders feeling that the project is not producing tangible and quick impact	[10, 22, 26, 29, 33]	SC6.1	Poor awareness of the potential benefits of the project
	PC6		[10, 22, 26, 29, 33]	SC6.2	Failure to produce tangible interim results
	PC7	Lack of explicit request from simulation group for active participation of stakeholders	[21]	SC7.1	Stakeholders’ high expectations
				SC7.2	Possible conflicts amongst various stakeholder groups
				SC7.3	No consultancy fees charged
	PC8	Poor management support	[10, 26]	SC8.1	Budgetary control requirements and fear of overspending
	PC8	Poor management support	[10, 26]	SC8.2	Management not involved in the governance of project
	PC9	Weak stakeholder “motivation” for involvement	[23]	SC9.1	Low competition in the industry
				SC9.2	Very few plans for improvement in the organization
				SC9.3	Poor understanding of stakeholder’s needs and expectations
	PC10	Skepticism towards classic management approaches	[22, 26, 31]	SC10.1	Cases of failure in delivering real benefits to the healthcare services in the past
				SC10.2	Morally unacceptable
				SC10.3	Low-profile reputation of modelers
Technical	PC11	Difficulties with understanding and working with simulation tools, techniques and models	[10, 18]	SC11.1	Poor interactivity
				SC11.2	User-unfriendliness of tools or techniques
				SC11.3	Poor conceptual modeling
	PC12	Unacceptable results for stakeholders	[10]	SC12.1	Poor presentation of results
				SC12.2	Incorrect or biased models
				SC12.3	Incomplete models
				SC12.4	Poor model flexibility
Project management	PC13	Lengthy project	[32]	SC13.1	Lengthy activities
	PC13	Lengthy project	[32]	SC13.2	Poor time management
	PC14	Too expensive, or cost-ineffective project	[10, 25, 28]	SC14.1	Too much work for a simple solution
	PC15	Failure to meet project objectives	[10]	SC15.1	Ill-defined objectives
	PC15	Failure to meet project objectives	[10]	SC15.2	Lack of focus on objectives
	PC16	Poor team efforts from either side	[29]	SC16.1	Weak teamwork from simulation modelers side
				SC16.2	Weak teamwork from stakeholders side
				SC16.3	Not enough communication between two sides
Health industry-specific	PC17	Healthcare problems are very complex to model	[27, 30, 31, 32]	SC17.1	Much variations involved
				SC17.2	Different views from a large number of different stakeholders involved
				SC17.3	Dynamic and multi-dimensional nature of healthcare
	PC18	Confidentiality issues	[22]	SC18.1	Confidentiality of patient data and/or ethical issues

The sources supporting the secondary causal factors are available upon request.

In order to draw implications for research and practice, there is a need for a robust evaluation of our categorization and hierarchical ordering of the causal factors (18 primary and 44 secondary), as shown in Table 1. In this regard and considering that domain experts are a reliable source of informing decision-making models/frameworks concerning the issues for which prior data are scarce,³⁵ we investigate opinions of experts who are familiar with and experienced in terms of both the method, that is, simulation, and the context, that is, healthcare systems.

The primary objective of this study is to elicit domain experts’ opinions, which works as a means to test the research hypotheses about the causal factors at two hierarchical levels, that is, primary and secondary. The secondary objective is to investigate the associations between various causal factors and to determine the contribution, explained as the effect size, of secondary causal factors (SCs) in explaining the variance in the primary causal factors (PCs).

3. Methods

3.1 Cause-and-effect analysis

The concepts of cause and effect, causal factors, and root cause approach used in this study are inspired by the work of Ishikawa.³⁶ Anything that affects an effect, either directly or indirectly, is defined as a causal factor of that effect. The root cause approach tends to identify causal factors in a hierarchical order that moves us towards the root causes of an issue. Figure 1 exemplifies the approach adopted in this research in two hierarchical levels.

3.2 Expert opinion survey

3.2.1 Participants

The target population aimed at domain experts having both “knowledge” of and “experience of involvement” in healthcare simulation projects. As one important aspect of this research concerned about learning from past experiences in real cases, especially from the users’ point of view, we welcomed opinions from healthcare professionals who have been involved in healthcare simulation projects, but with basic knowledge of simulation modeling and its applications. In order to ensure that these criteria are met, several steps were followed. Firstly, three types of organizations as the sources of potential experts were identified: healthcare provider organizations, simulation software/consulting companies, and academic institutes. Then, three search methods were selected that helped us to nominate 91 potential experts. These methods were (a) search of literature for people involved in healthcare simulation projects, (b) the member directory of a network of simulation modelers and practitioners in healthcare called MASHnet (www.mashnet.info), and (c) the authors’ previous contacts through a number of related projects. These 91 people came from a variety of professions but all were assumed to meet the “knowledge” and “experience” criteria. Finally, two questions were designed in the questionnaire to evaluate the respondents’ simulation knowledge and experience in the healthcare contexts, as a result of which, one response was removed as representing one with no experience. The respondents were informed about the survey topic, namely “healthcare simulation”, and the survey objectives in the Introduction to the questionnaire.

3.2.2 Questionnaire development

Using an online survey development software and data collection tool called SurveyMonkey (www.surveymonkey.com), we developed a questionnaire that consisted of three sections as follows.

Introduction: included information about the survey, the study objectives, a quick guide about how to complete the questionnaire, information on ethical issues/approval, and introduction of the research team and their contact details.

About you: this section comprised eight questions that asked information about the respondent’s characteristics, including the type of organization and country they were currently working in, level of knowledge about simulation, level of experience in healthcare simulation, and contact details (as optional).

Main topic of the survey: this section was the main part of the survey where experts were asked for their opinions about causal factors at two hierarchical levels (i.e., 18 primary and 44 secondary factors), as presented in Table 1. A balanced five-point Likert scale format (“strongly agree”, “agree”, “neutral”, “disagree”, “strongly disagree”) plus a “don’t know” option was used to capture respondents’ opinions. An example of questions can be seen in Figure 2. In addition, open-ended questions and spaces were provided to allow respondents to express their opinions on more freely. A copy of the questionnaire is available on request from the authors.

3.2.3 Piloting

The questionnaire was pilot tested and checked for clarity and consistency by six experts in healthcare qualitative studies. Suggestions about better articulation of factors and also about allowing respondents to express their opinions in an open-ended format were received and incorporated in the questionnaire.

3.2.4 Launch and implementation of the survey

The survey was launched in November 2011 for a period of two weeks. The experts were invited to the survey by an email. Ethics approval for this study was obtained from the relevant ethics committee. We did not seek the written consent from the experts for participating in the study; however, the return of a completed questionnaire was considered as a valid consent of the individual participant.

3.3 Data analysis methods

Data were processed using the software IBM^® PAWS Statistics 18. Participants’ responses were coded as “strongly agree” = 1, “agree” = 2, “neutral” = 3, “disagree” = 4, “strongly disagree” = 5, and “don’t know” = 0. Cronbach’s alpha was used for checking the reliability and internal consistency of each causal factor comprising a primary cause (PC) and its secondary causes (SCs). Descriptive statistics (i.e., mean [M], median [Med], mode [Mod], and standard deviation [SD]) were determined on the original sample, which was relatively small (n = 37). The bootstrapping approach is useful for mitigating the effects of small sample size and lack of normal distribution of data, and making reliable statistical inferences.^37,38 We therefore used the bootstrapping method as reported. We compared the descriptive statistics obtained with the original sample (n = 37) with the descriptive statistics obtained with a hypothetical larger sample equal to 1000 using bootstrap resampling with 1000 iterations.³⁹ We used the median values for drawing the inferences because this is a better indicator compared to the mean value, especially when the data distribution is not normal.⁴⁰ In fact, our data, which was tested by the Kolmogorov–Smirnov test and Shapiro–Wilk tests, revealed deviation from normal distribution. We therefore used nonparametric Kendall’s tau correlations, which are more suitable for small sample size and not-normally distributed data,^40,41 for assessing the relationship between PCs, between SCs, and between SCs and PCs, using both the original and a bootstrapped sample with 1000 iterations, as reported by others.³⁹

Another possibility of checking relationships between the causal factors could be determined by running factor analysis, but we could not run this kind of test due to limitations of our data, that is, small sample size and non-normal distribution. We did not transform the data to meet the normalized data distribution assumption necessary for the multiple linear regression models.⁴¹ We therefore run non-linear multiple regression models to determine the effect size of each SC in explaining the variance in its hypothesized PC and possibly with other PCs to which it was statistically significantly correlated (p < 0.05). In non-linear multiple regression modeling, a PC was used as the outcome/dependent variable while all SCs hypothesized with it were used as independent/predictor variables, which were entered simultaneously in the regression model to obtain the non-linear regression estimates. Then, the effect size analysis obtained on the original sample was checked by non-linear regression estimates obtained on a bootstrapped sample of 1000 experts.

Finally we used a 2×2 matrix, as inspired by the widely known Product Portfolio Model,⁴² as a way to map the spectrum of simulation modeling applications in healthcare alongside the findings of this research, and to offer insights and to instigate further discussions on this topic.

4. Results and analysis

Forty one out of the 91 experts responded to our survey, making an overall response rate of 45%. However, four of the experts completed only the “About You” section of the questionnaire, one of whom expressed no experience in having been involved in a healthcare simulation project. Therefore, only 37 forms were considered for the data analysis, making an effective response rate of about 41%.

Questions about specific causal factors in Section C were answered by 78% or more of the respondents and the average response rate to these questions was 90%. The results of the questionnaire survey are presented in Table 2.

Table 2.

Survey data and descriptive statistics about primary and secondary factors, based on both the original and bootstrapped data.

	Survey data							Descriptive statistics
Factor ID	1 = Strongly agree (n)	2 = Agree (n)	3=Neutral (n)	4 = Disagree (n)	5 = Strongly disagree (n)	Do not know (n)	Missing (n)	Original sample (n = 37)				Bootstrapped sample (n = 1000)				Decision
Factor ID	1 = Strongly agree (n)	2 = Agree (n)	3=Neutral (n)	4 = Disagree (n)	5 = Strongly disagree (n)	Do not know (n)	Missing (n)	Mean	Med	Mod	SD	Mean	Med	Mod	SD	Retained (R) Excluded (E)
PC1	2	14	7	13	1	0	0	2.92	3	2	1.04	3.26	4	4	0.99	E
SC1.1	0	19	2	15	1	0	0	2.95	2	2	1.03	2.99	3	2	1.01	E
SC1.2	1	12	9	9	2	0	4	2.97	3	2	1.02	3.12	3	4	1.02	E
PC2	14	14	6	3	0	0	0	1.95	2	1^a	0.94	1.91	2	1	0.93	R
SC2.1	1	11	7	15	0	3	0	2.81	3	4	1.24	2.74	3	4	1.29	E
SC2.2	8	4	10	7	0	3	5	2.31	3	3	1.33	2.3	3	3	1.35	E
PC3	17	10	2	5	2	0	1	2.03	2	1	1.28	2.03	2	1	1.25	R
SC3.1	13	17	4	2	0	0	1	1.86	2	2	0.83	1.89	2	2	0.83	R
SC3.2	12	13	4	2	1	0	5	1.97	2	2	1.03	2.01	2	2	1.01	R
PC4	3	19	9	5	1	0	0	2.51	2	2	0.93	2.53	2	2	0.89	R
SC4.1	4	16	9	4	2	1	1	2.47	2	2	1.11	2.44	2	2	1.06	R
SC4.2	9	16	5	2	1	1	3	2.03	2	2	1.03	2.03	2	2	1.06	R
PC5	13	22	2	0	0	0	0	1.7	2	2	0.57	1.73	2	2	0.56	R
SC5.1	13	16	4	4	0	0	0	1.97	2	2	0.96	2.01	2	2	0.90	R
SC5.2	5	18	9	4	0	0	1	2.33	2	2	0.86	2.3	2	2	0.80	R
SC5.3	8	24	2	2	0	1	0	1.89	2	2	0.77	1.89	2	2	0.69	R
SC5.4	5	17	8	5	0	0	2	2.37	2	2	0.91	2.39	2	2	0.82	R
PC6	9	16	6	3	0	1	2	2.03	2	2	0.95	2.03	2	2	1.04	R
SC6.1	13	11	5	6	0	1	1	2.06	2	1	1.15	2.14	2	1	1.17	R
SC6.2	6	16	8	3	0	1	3	2.18	2	2	0.94	2.14	2	2	0.93	R
PC7	3	9	8	10	3	2	2	2.86	3	4	1.33	2.7	3	2	1.40	E
SC7.1	1	7	14	10	1	1	3	3	3	3	1.02	2.9	3	3	1.04	E
SC7.2	5	15	10	2	0	2	3	3.09	3	3	1.11	3.01	3	3	1.11	E
SC7.3	1	1	15	12	0	5	3	2.82	3	3	1.36	2.76	3	3	1.34	E
PC8	6	14	9	2	0	2	4	2.09	2	2	0.98	2.11	2	2	1.02	R
SC8.1	3	10	10	6	0	3	5	2.41	2.5	2^a	1.19	2.34	2	3	1.20	R
SC8.2	4	13	7	6	0	2	5	2.34	2	2	1.13	2.26	2	2	1.10	R
PC9	4	17	7	5	1	1	2	2.4	2	2	1.06	2.48	2	2	1.11	R
SC9.1	0	12	8	9	0	4	4	2.55	3	2	1.25	2.56	3	2	1.26	E
SC9.2	0	7	7	16	1	2	4	3.15	4	4	1.18	3.24	4	4	1.16	E
SC9.3	1	17	8	4	0	3	4	2.27	2	2	1.04	2.24	2	2	1.05	R
PC10	5	16	10	5	0	0	1	2.42	2	2	0.91	2.43	2	2	0.79	R
SC10.1	6	18	3	4	1	3	2	2.06	2	2	1.16	1.99	2	2	1.10	R
SC10.2	1	4	7	16	5	1	3	3.5	4	4	1.16	3.56	4	4	1.18	E
SC10.3	0	8	11	8	2	4	4	2.76	3	3	1.35	2.7	3	3	1.40	E
PC11	11	15	1	7	0	0	3	2.12	2	2	1.09	2.14	2	2	1.12	R
SC11.1	3	14	7	7	0	0	6	2.58	2	2	0.96	2.64	2	2	0.94	R
SC11.2	7	16	4	5	0	0	5	2.22	2	2	0.98	2.22	2	2	0.95	R
SC11.3	7	11	3	10	0	0	6	2.52	2	2	1.18	2.62	2	4	1.19	R
PC12	0	12	6	16	0	1	2	3.03	3	4	1.04	3.02	3	4	1.07	E
SC12.1	3	11	9	7	0	1	6	2.58	3	2	1.06	2.64	3	2	1.01	E
SC12.2	0	12	13	6	0	1	5	2.72	3	3	0.89	2.73	3	3	0.89	E
SC12.3	2	13	11	4	1	1	5	2.56	2.5	2	1.01	2.59	3	2^a	1.09	E
SC12.4	3	13	12	3	0	1	5	2.41	2	2	0.91	2.41	2	2	0.95	R
PC13	6	18	6	5	0	0	2	2.29	2	2	0.93	2.3	2	2	0.93	R
SC13.1	4	16	7	5	0	0	5	2.41	2	2	0.91	2.46	2	2	0.94	R
SC13.2	3	12	12	3	0	1	6	2.42	2	2^a	0.92	2.47	3	3	0.99	E
PC14	2	16	10	6	0	1	2	2.66	2	2	0.94	2.64	2	2	0.90	R
SC14.1	1	14	11	2	1	1	7	2.5	2	2	0.94	2.47	2	2	0.83	R
PC15	0	14	12	7	0	1	3	2.71	3	2	0.91	2.66	3	2	0.93	E
SC15.1	8	11	9	1	0	1	7	2.03	2	2	0.93	2.02	2	2	0.92	R
SC15.2	4	17	6	1	1	1	7	2.17	2	2	0.95	2.16	2	2	0.96	R
PC16	1	10	15	7	0	1	3	2.76	3	3	0.92	2.83	3	3	0.89	E
SC16.1	0	5	14	9	0	1	8	3.03	3	3	0.91	3.09	3	3	0.87	E
SC16.2	1	7	16	4	0	1	8	2.72	3	3	0.88	2.79	3	3	0.84	E
SC16.3	4	12	7	5	0	1	8	2.38	2	2	1.05	2.47	2	2	1.07	R
PC17	10	18	1	4	0	1	3	2.06	2	2	1.04	2	2	2	1.01	R
SC17.1	11	16	1	2	1	0	6	1.9	2	2	0.98	1.84	2	2	0.94	R
SC17.2	12	9	8	2	0	0	6	2	2	1	0.97	2.02	2	1	0.90	R
SC17.3	9	15	0	3	2	1	7	2.03	2	2	1.22	1.94	2	2	1.15	R
PC18	3	8	6	12	4	1	3	3.09	3	4	1.31	3.04	3	4	1.28	E
S.C18.1	4	8	6	9	3	1	6	2.87	3	4	1.34	2.89	3	2	1.32	E

Multiple modes exist. The smallest value is shown.

Rank codes: 1 = strongly agree, 2 = agree, 3 = neutral, 4 = disagree, 5 = strongly disagree.

4.1 Descriptive statistics

The descriptive statistics for both the original sample and the bootstrapped sample are presented in Table 2. We decided either to “retain” or to “exclude” any of the PCs and SCs based on the following protocol.

Retain a causal factor: the PCs and SCs with a median rank from 1 (strongly agree) to 2 (agree) should be retained for further analyses.

Exclude a causal factor: all PCs and SCs with a median rank from 3 (neutral) to 5 (strongly disagree) should be excluded from further analyses.

Accordingly, 6 PCs and 18 SCs had the mean and median rankings ⩾3, which suggested that the experts were either neutral (rank = 3) or disagreed (disagree = 4 and strongly disagree = 5) about these PCs and SCs being causal factors of low stakeholder engagement in healthcare simulation projects. On the other hand, 12 PCs and 26 SCs received the mean and median rankings <3 (i.e., strongly agree = 1 and agree = 2), which suggested that among the tested factors (Table 1) the experts believed that these were the only reliable causal factors of low stakeholder engagement in healthcare simulation projects. The factors that were found reliable, hence retained, and those that were found unreliable, hence excluded, are marked as R and E respectively in the “Decision” column in Table 2.

In the following sections, we present our findings related to only the reliable/retained (R) PCs and SCs.

4.2. Correlation analysis

4.2.1 Correlations between retained PCs

The nonparametric Kendall’s tau-b (two-tailed) correlation analysis revealed five significant correlations (p < 0.05) between eight out of 12 retained PCs (Table 3), while the remaining PCs were not significantly correlated with any other PC.

Table 3.

Nonparametric correlations (Kendall’s tau-b) between primary causal factors (PCs).

Primary factors	Correlation coefficient	Significance level (p)
PC3 ↔ PC11	0.395	0.01
PC5 ↔ PC11	0.356	0.026
PC6 ↔ PC9	0.321	0.031
PC8 ↔ PC9	0.312	0.037
PC14 ↔ PC17	0.352	0.021

4.2.2 Correlations between retained SCs

Results of nonparametric correlations (i.e., Kendall’s tau-b [two-tailed]) between retained SCs (n = 26) revealed statistically significant associations between the majority of SCs belonging to the same group (Table 4).

Table 4.

Nonparametric correlations (Kendall’s tau-b) between retained secondary causal factors (SCs).

Secondary factors	Correlation coefficient	Significance level (p)
SC3.1 ↔ SC3.2	0.746	0.001
SC4.1 ↔ SC4.2	0.406	0.006
SC5.1 ↔ SC5.2	0.483	0.001
SC11.1 ↔ SC11.2	0.571	0.000
SC11.1 ↔ SC11.3	0.385	0.015
SC15.1 ↔ SC15.2	0.661	0.001
SC17.1 ↔ SC17.2	0.530	0.001
SC17.1 ↔ SC17.3	0.543	0.001
SC17.2 ↔ SC17.3	0.602	0.001

SCs that were not statistically significantly associated with other SCs within the same group were SC5.3, SC5.4, SC6.1, SC6.2, SC8.1, SC8.2, SC9.3, and SC13.1. There were, however, numerous statistically significant associations between SCs belonging to different groups (not reported here but available on request from the authors).

4.2.3 Correlations between retained SCs and retained PCs

Statistically significant associations between retained SCs and retained PCs are presented in Table 5. Results showed that except for SC5.3, SC5.4, SC8.1, SC9.3, SC11.3, SC12.4, SC15.1, SC15.2, and SC16.3, all other SCs were statistically significantly associated with the PC within the hypothesized group. In addition, there were statistically significant associations between SCs and PCs belonging to different groups (Table 5). It is noteworthy that SC5.4, SC9.3, SC11.3, and SC12.4 were not statistically significantly associated with any PC.

Table 5.

Nonparametric correlations (Kendall’s Tau-b) between secondary causal factors (SCs) and primary causal factors (PCs).

Secondary causal Factors	Primary causal factors
Secondary causal Factors	2	3	4	5	6	8	9	10	11	12	13	14	15	16	17
3.1		.637**							.501**			.309*
3.2		.696**	−.334*	.331*					.400*
4.1			.663**			.320*
4.2			.368*
5.1				.586**					.420**			.378*
5.2				.384*					.383*			.328*
5.3^a					.344*
5.4^a,b
6.1					.476**		.505**		.431**
6.2					.497**
8.1^a			.304*								−.446**
8.2						.493**
9.3^a,b
10.1								.417**				.445**			.384*
11.1									.577**
11.2									.686**
11.3^a,b
12.4^a,b
13.1											.796**
14.1					.353*							.419*
15.1^a		.334*					.337*
15.2^a		.323*				.325*
16.3^a	.391*
17.1												.435**			.662**
17.2												.461**			.567**
17.3									.356*			.355*			.692**

Notes: empty cells represent non-significant correlations. ^a SCs not associated with hypothesized PCs; ^b SCs not associated to any PCs. * Correlation is significant at the 0.05 level (two-tailed). **Correlation is significant at the 0.01 level (two-tailed).

4.3 Extraction of the variance by SCs in PCs

4.3.1 Non-linear multiple regression using original and bootstrapped samples

Results of non-linear multiple regression for determining explanation of the variance by SCs in PCs are presented in Table 6. The results from the original sample showed that 17 retained SCs explained a significant percentage of the variance in the PC of the hypothesized group and the scale reliabilities of retained SCs and PC belonging to the same group were α⩾.600. Five SCs—5.3, 5.4, 8.1, 11.3, and 12.4—did not explain any significant variance in the PCs of hypothesized groups. In addition, several SCs explained statistically significant variance in PCs of other groups; however, the scale reliabilities of such alternative associations were low (α < .600) in most of the cases (Table 6). Results of bootstrapped non-linear regression comprising only the R² statistics are also presented in Table 6, which confirm that most of the SCs statistically significantly explained the variance in the PC of the own group and the PCs of other groups. Nevertheless, the magnitude of the variance explained by SCs in PCs outside the hypothesized groups was mostly lower compared to the amount of variance explained in the PC of the hypothesized group.

Table 6.

Results of non-linear multiple regression using original and bootstrapped samples.

Predictor variables	Dependent variables	Reliability analysis	Original sample							Bootstrapped sample
SC	Hypothesized PC	Cronbach’s alpha	b	SEb	beta	F	P	R ²	Adjusted R²	R ²
3.1	3	.845	0.640	0.039	0.521	52.534	0.001	0.425	0.417	0.403
3.2	3	.845	0.628	0.022	0.428	181.771	0.001	0.719	0.715	0.759
4.1	4	.836	0.779	0.019	0.456	107.842	0.001	0.617	0.611	0.666
4.2	4	.836	0.919	0.034	0.762	5.356	0.024	0.074	0.060	0.098
5.1	5	.770	0.757	0.027	0.507	60.580	0.001	0.460	0.453	0.537
5.2	5		0.778	0.033	0.563	34.909	0.001	0.330	0.320	0.355
5.3^a	5						ns
5.4^a	5						ns
6.1	6	.716	0.788	0.034	0.574	29.857	0.001	0.308	0.298	0.376
6.2	6	.716	0.821	0.051	0.695	10.244	0.002	0.133	0.120	0.178
8.1^a	8	.610					ns
8.2	8	.610	0.815	0.041	0.632	16.846	0.001	0.211	0.198	0.235
9.3	9	.600	0.711	0.042	0.550	34.041	0.001	0.358	0.348	0.413
10.1	10	.790	0.830	0.034	0.604	20.806	0.001	0.254	0.242	0.261
11.1	11	.875	0.696	0.034	0.515	56.006	0.001	0.441	0.433	0.389
11.2	11		0.638	0.021	0.425	193.663	0.001	0.732	0.728	0.755
11.3^a	11						ns
13.1	13	.899	0.678	0.021	0.436	157.568	0.001	0.689	0.685	0.707
14.1	14	.601	0.850	0.041	0.687	11.313	0.001	0.141	0.128	0.131
17.1	17	.891	0.737	0.030	0.514	56.740	0.001	0.444	0.436	0.657
17.2	17		0.716	0.035	0.536	45.315	0.001	0.390	0.381	0.307
17.3	17		0.767	0.029	0.521	49.394	0.001	0.424	0.416	0.561
	Alternative PCs
3.1	11	.628	0.721	0.047	0.599	25.299	0.001	0.263	0.252	0.229
3.1	14	.301	0.890	0.039	0.738	7.212	0.009	0.092	0.079	0.070
3.2	5	—					ns
3.2	11	.500	0.803	0.044	0.651	16.060	0.001	0.184	0.173	0.130
4.1	8	.450	0.864	0.043	0.709	8.470	0.005	0.119	0.105	0.153
5.1	11	.600	0.768	0.047	0.632	18.934	0.001	0.211	0.199	0.172
5.1	14	.592	0.862	0.032	0.652	15.929	0.001	0.183	0.172	0.182
5.2	11	.533	0.706	0.042	0.566	34.038	0.001	0.324	0.315	0.265
5.2	14	—					ns
5.3	6	.700	0.671	0.041	0.536	42.740	0.001	0.389	0.380	0.439
6.1	9	.667	0.741	0.024	0.470	88.965	0.001	0.570	0.564	0.573
6.1	11	—					ns
10.1	14	.601	0.876	0.039	0.703	8.641	0.005	0.124	0.110	0.095
10.1	17	.595	0.865	0.054	0.752	5.384	0.024	0.081	0.066	0.028
14.1	6	.522	0.765	0.052	0.644	15.651	0.001	0.194	0.182	0.250
15.1	3	—					ns
15.1	9	—					ns
15.2	3	—					ns
15.2	8	.523	0.848	0.045	0.695	9.633	0.003	0.133	0.119	0.176
16.3	2	—					ns
17.1	14	.574	0.909	0.034	0.748	6.556	0.013	0.085	0.072	0.059
17.2	14	.654	0.863	0.036	0.679	12.554	0.001	0.150	0.138	0.120
17.3	11	.479	0.822	0.044	0.664	13.521	0.001	0.168	0.155	0.157
17.3	14	—-					ns

ns: not significant. ^a Secondary causal factors (SCs) not associated with hypothesized primary causal factors (PCs).

4.4 Reliability analysis of retained PCs and associated SCs

The reliability analysis of each retained PC with its associated SCs (presented in Table 6) revealed that all these retained factors have a good Cronbach’s alpha (α ⩾ .700), except for PCs 8, 9, and 14 with their related SCs that have a slightly lower but a respectable reliability (α ⩾ .600) (For the interpretation of α levels see Davis,⁴³ Nunnally,⁴⁴ and Peterson⁴⁵.)

5. Discussion and implications

Despite the fact that this survey addressed a specific area and that the questionnaire was long, the average response rate to the questions about causal factors was high (about 90%). This implies that our survey was well received by the respondents.

5.1 Primary causal factors

Our results showed that 12 PCs out of 18 PCs were reliable, which we named as retained PCs (Table 2). Table 7 presents the top three statistically significant PCs based on experts’ ratings with bootstrapped mean and median values of ⩽2. Among these PCs, the “Communication gap between simulation and stakeholder groups (PC5)” emerged as the strongest causal factor on the basis of the mean ranking (i.e., 1.73 ± 0.56) (Table 2). In addition, all its SCs received significant support from the experts. This finding provides further evidence in support of a number of previous studies, which reported that communication between analysts and users is often problematic due to issues such as cognitive limitations and vocabulary differences.^16,46–48 While the communication gap is arguably a universal issue in all sectors, healthcare seems to suffer more from this due to wider background differences between the management and the clinical communities. As reflected in the SCs associated with the PC5, some part of the communication gap problem also pertains to the technology-intensive nature of simulation. This can be supported by the results of correlation analysis shown in Table 3, which suggest that there exists a possible link with technical factors represented by PC11. This finding highlights the importance of continuing efforts that have been going on to make simulation tools and techniques simple and understandable by the users.

Table 7.

Top three statistically significant primary causal factors (PCs; mean and median ratings ≤ 2).

#	Code	Primary causal factor
1	PC5	Communication gap between simulation and stakeholder groups
2	PC2	Stakeholder’s (e.g., clinician’s) high workload
3	PC17	Healthcare problems are very complex to model

Our findings also show that “Stakeholders high workload (PC2)” and “too much complexity in the healthcare settings (PC17)” are two other significant primary factors that could deter stakeholder engagement in simulation projects.

On the other hand, analysis of the results revealed that experts were either neutral or disagreed about six PCs as being the primary causes (denoted as excluded in Table 2). Further investigation may be required for some of these excluded PCs. For example “reluctance to change (PC1)”, which seems to be a common feature of many change projects including simulation ones, could not receive enough support as being a causal factor from the experts in our study. Whether this is an observation associated with the healthcare contexts remains to be explored. As one of the respondents puts it “Stakeholders, especially clinicians, have an urgent desire and enthusiasm for change, but have become cynical and disenchanted about the way in which change is conceived, implemented and evaluated”.

The other factor is “ethical issues” about which there were mixed opinions, although the experts overall found it less of a real barrier. As two respondents stated, it is more of a “red herring” in reality.

Exclusion of PC15 (i.e., “failure to meet objectives”) might reflect poor articulation in the questionnaire, because both of its hypothesized SCs received very significant expert support. Therefore, we suggest that a new PC called “Inadequate definition and affirmation of objectives”, which would represent both SC15.1 and SC15.2, replaces PC15.

5.2 Secondary causal factors

Table 8 presents the top five statistically significant SCs supported by the experts’ ratings. This is based on bootstrapped mean and median values of ⩽2, as outlined earlier in Table 2. As SCs are meant to be closer to the roots of the issue, careful attention needs to be given to these factors, especially to the ones with the most statistically significant ratings, as listed in the Table 8.

Table 8.

Top five statistically significant secondary causal factors (SCs; mean and median ratings ≤2).

#	Code	Secondary causal factor
1	SC17.1	Much variations involved
2	SC5.3	Irregular contacts between simulation and stakeholder groups
3	SC3.1	Stakeholders’ low exposure to simulation projects
4	SC17.3	Dynamic and multi-dimensional nature of healthcare
5	SC10.1	Cases of failure in delivering real benefits to the healthcare services in the past

An important observation is the retention of some SCs associated with an excluded hypothesized PC. In one example, as suggested earlier in the previous section, two SCs, that is, “ill-defined objectives (SC15.1)”, and “lack of focus on objectives (SC15.2)” merged to create a new PC that replaced PC15. Similarly, in the case of SC12.4 (Poor model flexibility), it was converted to a new PC that replaced PC12. However, in the case of SC16.3, i.e., “Not enough communication between two sides”, it was merged with the very closely related SC5.3 (Irregular contacts between two groups) to create a new SC called as “Irregular or insufficient contacts between two groups”.

5.2.1 Extraction of the variance in PCs by SCs

According to Field,⁴¹ the correlation coefficient can be used to determine the effect size in which the coefficient values of ±0.1 (i.e., 10%), ±0.3 (i.e., 30%), and ±0.5 (i.e., 50%) represent a small, medium, and large effect size, respectively. Looking at the findings of extraction of the variance in PCs by SCs (Table 6), we believe that retained SCs (Table 2) can be classified into four categories on the basis of the level of the variance explained in any retained PC (Table 6), as described below.

5.2.1.1 Strong SCs

All those SCs that statistically significantly explain 30% or more variance in a hypothesized and/or an alternate PC. Consequently, this group of SCs includes SC3.1, SC3.2, SC4.1, SC5.1, SC5.2, SC6.1, SC9.3, SC11.1, SC11.2, SC13.1, SC17.1, SC17.2, and SC17.3.

5.2.1.2 Moderate SCs

All those SCs that statistically significantly explain 10–30% of the variance in a hypothesized and/or an alternate PC. This group comprises SC6.2, SC8.2, SC10.1, and SC14.1.

5.2.1.3 Weak SCs

All SCs that statistically significantly explain less than 10% of the variance in a hypothesized and/or an alternate PC. This group includes only SC4.2.

5.2.1.4 Independent SCs

All SCs that received experts’ agreement as being a possible SC but they could not explain any statistically significant variance in any PCs. This group comprises SC5.4, SC11.3, SC12.4, SC15.1, SC15.2, and SC16.3. Amongst these, SC5.4, SC11.3, and SC12.4 were converted to three new independent PCs. On the other hand, SC15.1 and SC15.2 were merged together, creating a new PC, because these two SCs were strongly correlated with each other. Finally, SC16.3 was merged with SC5.3 to create a new SC, as explained earlier.

Figure 3 shows a cause-and-effect diagram that represents the causal relationships in two levels with inputs from the analysis of our results as explained above. In addition, six extra links shown as “line curves” were added between a number of SCs and alternative PCs. This is a result of the analysis carried out in the bottom section of Table 6 with regards to the associations with reliabilities α ⩾ .600.

5.3 Implications for healthcare simulation projects

As SCs are meant to be closer to the roots of the issue, careful attention needs to be given to these factors, and specifically to the most significant ones (Table 8). With regards to two secondary factors in Table 8, namely “Much variations involved” and “Dynamic and multi-dimensional nature of healthcare”, we believe that although there is a high level of variations, dynamism, and complexity inherent in the healthcare systems,³⁰ the healthcare sector represents a wide spectrum of complexity levels in the same way as other sectors do. On one side of the spectrum, there are areas that are characterized by fewer complexities in the sense that they may require less of system details or modeling of interactions and more of aggregate data, which are mostly available in public domains. Areas such as prevalence studies of one specific disease from the public health perspective belong to this side. These areas are normally covered by the SD method. The other side of the spectrum, however, can represent areas that are concerned with individuals (patients, clinicians, staff, and others) and modeling their behaviors and interactions within a larger system such as a hospital, a local community, or a country. An implication of our findings could be that if the less complex side of the spectrum, such as the public health applications, receives a higher priority and/or publicity, an increasing level of stakeholder involvement, and subsequent success in healthcare would be realized. Such a general approach tends to handle high variations (SC17.1 or causal factor one in Table 8) and dynamism (SC17.3 or causal factor four in Table 8), through controlling the scale of datasets and details of the project. This is analogous to the discussion of “risk management” in a project where risks are reduced as a result of streamlining the project. Furthermore, an increased rate of success that follows the streamlining efforts will bring in a domino effect to the future projects, hence addressing the “historical failure” factor (SC10.1 or causal factor five in Table 8).

Another implication, which relates to the factor three in Table 8 (low exposure to simulation or SC3.1) would be to give higher priority to addressing more widespread problems with high impacts such as public health scenarios or operational management of general hospitals and emergency departments, although looking from a wider and system’s perspective. This will result in a wider exposure of healthcare systems and stakeholder community (especially clinicians), either directly or indirectly, to the benefits and impacts of simulation modeling projects.

These implications reaffirm the current thoughts in the simulation community by providing more logical backing of the idea. We believe that the findings of the current study recognize the vital susceptibility of healthcare simulation modeling on two dimensions, namely “complexity” and “impact”. Therefore, we provide further insights into these two dimensions by presenting a 2×2 matrix model view, as inspired by the widely known Product Portfolio Model,⁴² with “problem complexity” and “impact or stakeholder coverage” on each dimension of the matrix (Figure 4). Examples in the matrix represent types of application areas that could relate to these two dimensions. The top-left and bottom-right boxes in the matrix represent the areas with highest and lowest engagement generation in the long term, respectively. We believe that high-impact public health studies can exemplify the areas that will potentially generate engagement in the healthcare simulation domain.

We also suggest that a structured “stakeholder involvement plan” be developed and exercised throughout the project, which would address factor two (irregular contacts between stakeholder and simulation communities or SC5.3) and to some extent factor three (low exposure to simulation or SC3.1) in Table 8. The plan could incorporate key stakeholder groups and will involve various engagement models in the form of committees, workshops, or surveys that will take place on a regular basis. A stakeholder “champion” from within the healthcare organization could be selected to have a key role in the engagement activities. All these and other engagement-related arrangements could constitute the plan, which would be developed before a project starts.

6. Conclusions

While there is an evidence claiming that “low stakeholder engagement” has created a significant limitation on the achievement of simulation modeling practices in healthcare, an investigation of causal factors of “low stakeholder engagement” remains undeveloped.

Our expert opinion survey highlighted the significance of a number of these causal factors. Overall, “group communications”, “system complexity”, and “impact” were found to be the main areas that stakeholder engagement in healthcare simulation generally relate to. The implications of our findings highlight priorities that need to be given to the simulation applications with low complexity and high-impact/stakeholder coverage, such as high-impact public health studies and prevalence studies of a common disease.

Another significant finding of our study includes the identification of the factors that received experts’ disagreement, such as “reluctance to change”. This finding is useful in a way that it will serve, through further investigation, to contribute to cross-sectoral comparisons with an aim to evaluate contextual effects on the “resistance” factor. Ultimately, it could lead to rectifying some previous perceptions or adjusting existing widespread evidence^49–52 on the “resistance” factor.

The identification of causal factors can help simulation practitioners and project managers to avoid barriers in involving stakeholders; hence, this may increase the chance of a project’s success. As “stakeholder engagement” plays a rather universal role in many types of change projects, our survey methodology and instruments could make important contributions in providing the research community in the simulation field with a useful evidence as well as a means to conduct similar studies in other fields. Indeed, since stakeholder engagement is key to the success of so many other types of projects, there is a possibility of major impacts throughout many sectors, if generalizations can be made.

6.1 Study limitations and future research

The overall survey response rate was 41%, which seems moderate in the category of expert opinion surveys where the respondents (experts) are normally very busy, especially within the healthcare industry. Also, the survey sample size was relatively small, which was mainly due to a limited number of experts in this field. The sample size issue was, however, overcome to some extent by applying the bootstrapping methodology for generating a theoretical sample size of 1000 participants (using 1000 iterations), which is an accepted and reliable statistical technique that is widely used in research studies.

There is a need for further research involving another round of surveys on the basis of the findings of this study as well as further statistical tests, such as the exploratory and confirmatory factor analyses and structural equation modeling for finding the dimensions and the effect of each PC and SC. Also, further research could be conducted to bring in contextual parameters, such as public or private systems, primary or secondary care settings, etc. A larger sample of experts could be selected in the further studies.

We did three sets of correlation analyses, namely (a) between PCs, (b) between SCs and PCs, and (c) between SCs under each PC. A further study could also look at the correlations between SCs of different groups to explore whether there would be any new information with regards to the causal inter-relationships.

Although this study has focused on simulation, the authors believe that similar research into other types of projects might produce comparable conclusions.

Footnotes

Acknowledgements

The authors wish to thank the experts for taking part in this survey.

Funding

This work was supported by the Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH) program (EPSRC Grant EP/F063822/1).

Author biographies

Mohsen Jahangirian, PhD, has worked at Brunel University London, UK, for more than 6 years. His research interests are Operational Research, Healthcare Simulation, and Information Systems.

Simone Borsci, PhD, worked as a research fellow at Brunel University London. He is currently a research fellow on human factors on medical device at Imperial College of London for the Diagnostic Evidence Co-operative (DEC).

Syed Ghulam Sarwar Shah, MBBS, MSc, PhD, FRSPH, worked as a research fellow at the Brunel University London, UK, and currently he is a director of the Hi-DRs (Health Information, Data and Research Support) Ltd, UK.

Simon J E Taylor, PhD, MSc, BSc (Hons), is a reader in computing and leader of the Modelling & Simulation Group at Brunel University London, UK.

References

Joumard

André

Nicq

. Health status determinants: lifestyle, environment, health care resources and efficiency. OECD Economics Department Working Papers, No. 627, OECD Publishing.

Pannirselvam

Ferguson

Ash

. Operations management research: an update for the 1990s. J Oper Manag 1999; 18: 95-112.

Brailsford

Harper

Pitt

. An analysis of the academic literature on simulation and modelling in healthcare. J Simulat 2009; 3: 130-140.

Fone

Hollinghurst

Temple

. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Publ Health Med 2003; 25: 325–335.

Taylor

SJE

Eldabi

Riley

. Simulation modelling is 50. Do we need a reality check? J Oper Res Soc 2009; 60(SUPPL. 1): S69-S82.

Jahangirian

Taylor

SJE

Young

. Economics of modeling and simulation: Reflections and implications for healthcare. In: winter simulation conference, 2010.

Roberts

Maxwell

Gross

, Cost-effective care of end-stage renal disease: a billion dollar question. Ann Intern Med 1980; 92: 243–248.

Mustafee

Katsaliaki

Taylor

SJE

. Profiling literature in healthcare simulation. Simulation 2010; 86: 543–558.

Cooke-Davies

. The “real” success factors on projects. Int J Project Manag 2002; 20: 185-190.

10.

Robinson

Pidd

. Provider and customer expectations of successful simulation projects. J Oper Res Soc 1998; 49: 200–209.

11.

Wang

ETG

Chang

JYT

Jiang

JYJ

. User advocacy and information system project performance. Int J Project Manag 2011; 29: 146-154.

12.

Achterkamp

Vos

JFJ

. Investigating the use of the stakeholder notion in project management literature, a meta-analysis. Int J Project Manag 2008; 26: 749-757.

13.

Teo

TSH

Srivastava

Ranganathan

. A framework for stakeholder oriented mindfulness: case of RFID implementation at YCH Group, Singapore. Eur J Inform Syst 2011; 20: 201–220.

14.

Meyer

Stensaker

. Making radical change happen through selective inclusion and exclusion of stakeholders. Br J Manag 2009; 20: 219–237.

15.

Boonstra

Boddy

Bell

. Stakeholder management in IOS projects: analysis of an attempt to implement an electronic patient file, Eur J Inform Syst 2008; 17: 100–111.

16.

Alvarez

Urla

. Tell me a good story: using narrative analysis to examine information requirements interviews during an ERP implementation. DATA BASE Adv Inform Syst 2002; 33: 38–52.

17.

Shokri-Ghasabeh

Kavoousi-Chabok

. Generic project success and project management success criteria and factors: literature review and survey. WSEAS Trans Bus Econ 2009; 6: 456–468.

18.

Den Hengst

De Vreede

. Collaborative business engineering: a decade of lessons from the field. J Manag Inform Syst 2004; 20: 85-113.

19.

Levasseur

. People skills: marketing OR/MS - a people problem. Interfaces 2007; 37: 383–384.

20.

Levasseur

. People skills: ensuring project success - a change management perspective. Interfaces 2010; 40: 159-162.

21.

Melão

Pidd

. Use of business process simulation: a survey of practitioners. J Oper Res Soc 2003; 54: 2-10.

22.

Brailsford

Bolt

Connell

. Stakeholder engagement in health care simulation. In: winter simulation conference, 2009.

23.

Jahangirian

Naseer

Stergioulas

. Simulation in health-care: lessons from other sectors. Oper Res 2012; 12: 45–55.

24.

Freeman

. Strategic management: a stakeholder approach. Englewood Cliffs, NJ: Prentice-Hall, 1984.

25.

Eskinasi

Fokkema

. Lessons learned from unsuccessful modelling interventions. Syst Res Behav Sci 2006; 23: 483–492.

26.

Murphy

Perera

. Successes and failures in UK/US development of simulation. Simulat Pract Theor 2002; 9: 333–348.

27.

Jenkins

Deshpande

Davison

. Verification and validation and complex environments: a study in service sector. In: winter simulation conference, 1998.

28.

Brailsford

. Overcoming the barriers to implementation of operations research simulation models in healthcare. Clin Investig Med 2005; 28: 312–315.

29.

Harper

Pitt

. On the challenges of healthcare modelling and a proposed project life cycle for successful implementation. J Oper Res Soc 2004; 55: 657–661.

30.

Plsek

Greenhalgh

. The challenge of complexity in health care. Br Med J 2001; 323: 625–628.

31.

Lowery

Hakes

Keller

. Barriers to implementing simulation in health care. In: winter simulation conference, 1994.

32.

Jahangirian

Taylor

. RIGHT consolidated exemplar report. London: Brunel University, 2009.

33.

Eldabi

. Implementation issues of modeling healthcare problems: misconceptions and lessons. In: winter simulation conference, 2009.

34.

Jahangirian

Taylor

SJE

Eatock

. Causal study of low stakeholder engagement in healthcare simulation projects. J Oper Res Soc 2014; 66: 369–379.

35.

Kuhnert

. Four case studies in using expert opinion to inform priors. Environmetrics 2011; 22: 662–674.

36.

Ishikawa

. Guide to quality control. Tokyo: Asian Productivity Organization, 1982.

37.

Linden

Adams

Roberts

. Evaluating disease management program effectiveness: an introduction to the bootstrap technique. Dis Manag Health Outcome 2005; 13: 159-167.

38.

Fox

. Applied regression analysis and generalized linear models. 2nd ed. London: Sage, 2008.

39.

Akins

Tolson

Cole

. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Meth 2005; 5: 37.

40.

Polit

Beck

. Nursing research: generating and assessing evidence for nursing practice. 9th ed. Philadelphia, PA: Lippincott Williams & Wilkins, 2012.

41.

Field

. Discovering statistics using SPSS. 3rd ed. Los Angeles, CA: Sage, 2009.

42.

Henderson

. The product portfolio. BCG Perspectives 1970.

43.

Davis

. Educational measurements and their interpretation. Belmont, CA: Wadsworth, 1964.

44.

Nunnally

. Psychometric theory. 1st ed. New York: McGraw-Hill, 1967.

45.

Peterson

. A meta-analysis of cronbach’s coefficient alpha. J Consum Res 1994; 21: 381–391.

46.

Byrd

Cossick

Zmud

. A synthesis of research on requirements analysis and knowledge acquisition techniques. MIS Q 1992; 16: 117-138.

47.

Agarwal

Tanniru

. Knowledge acquisition using structured interviewing: an empirical investigation. J Manag Inform Syst 1990; 7: 123-140.

48.

Bostrom

. Successful application of communication techniques to improve the systems development process. Inform Manag 1989; 16: 279–295.

49.

Selander

Henfridsson

. Cynicism as user resistance in IT implementation. Inform Syst J 2012; 22: 289–312.

50.

Fiedler

. Managing resistance in an organizational transformation: a case study from a mobile operator company. Int J Project Manag 2010; 28: 370–383.

51.

Kim

Kankanhalli

. Investigating user resistance to information systems implementation: a status quo bias perspective. MIS Q Manag Inform Syst 2009; 33: 567–582.

52.

Doolin

. Power and resistance in the implementation of a medical management information system. Inform Syst J 2004; 14: 343–362.