E-learning of evidence-based health care (EBHC) to increase EBHC competencies in healthcare professionals: a systematic review

1.1.1 The need for evidence-based health care competencies

Evidence-based medicine (EBM), introduced in 1991, has its roots in the field of clinical epidemiology and was listed as “one of the 15 greatest medical milestones since 1840” in the British Medical Journal (Montori & Guyatt, 2008). The most commonly used definition of evidence-based medicine (EBM) describes it as “the conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients” (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). It thus requires practitioners to bring together external evidence that informs about the effects of new tests, treatments and interventions; clinical judgement and expertise of the clinician; and the patient's clinical state, values, preferences, needs and predicament.

These days, EBM is commonly referred to as evidence-based practice (EBP) or evidence-based health care (EBHC), as EBM is not limited to medical doctors, but should be adopted by all healthcare practitioners. Practicing EBHC typically involves five steps: i) Formulating an answerable question from a healthcare problem; ii) finding the best available evidence applicable to the question; iii) critically appraising the evidence for validity, clinical relevance and applicability; iv) applying the results of the evidence in the healthcare setting; and v) evaluating the performance (Dawes et al., 2005). An important aim of EBHC is that beneficial, effective health care practices are adopted and that harmful and ineffective ones are abandoned. Consequently, this requires healthcare professionals to recognise their deficiencies in knowledge and to adopt a philosophy of life-long learning, which is the backbone of practicing EBHC (Greenhalgh, 1997).

The importance of the knowledge, skills and attitude learnt through the principles of EBHC are also highlighted in the Lancet report on the health professional for the 21^st century (Frenk et al., 2010), which proposes that healthcare professional training should become transformative. One of the fundamental shifts of transformative learning aligns almost perfectly with the steps of EBHC; the shift from memorization of facts to “critical reasoning that can guide the capacity to search, analyse, assess and synthesise information for decision-making” (Frenk et al., 2010).

In addition, Glasziouand colleagues (2011) have urged educational institutions to teach medical students skills which enable them to become life-long learners so that they are able to combine external evidence from research with their own expertise and their patients’ values and preferences. They emphasize that teaching EBHC skills should form an integral part of the medical curriculum and be re-iterated and practiced throughout undergraduate and postgraduate training (Glasziou, Burls, & Gilbert, 2008; Glasziou, Sawicki, Prasad, & Montori, 2011).

In an evaluation of an online module of EBM, we have proposed a set of EBHC competencies that all healthcare professionals should ideally possess once they graduate (Rohwer, Young, & van Schalkwyk, 2013). These competencies comprise five key competencies that mirrorthe five steps of EBHC; and enabling competencies which include basic knowledge of epidemiology and biostatistics. Figure 1 is a graphic representation of the EBHC competencies.

OPEN IN VIEWER

Figure 1

EBHC enabling and key competencies

A number of systematic reviews have explored the effects of teaching EBHC to healthcare professionals, both at an under- and postgraduate stage. From the results, it is evident that teaching EBHC to students and health care professionals leads to increased EBHC knowledge and skills (Young, Rohwer, Volmink, & Clarke, 2014). The question is therefore no longer whether we should be teaching EBHC; but rather how we should be teaching it. Khan and Coomarasamy (2006) have proposed a hierarchy of teaching EBHC, where integrated and interactive teaching is seen as the most effective way of teaching EBHC (Khan & Coomarasamy, 2006). This, however, only refers to traditional lecture-based or face to face teaching of EBHC and does not include the increasingly popular method of electronic learning.

1.2.1 Electronic learning (e-learning)

Electronic learning or e-learning (as the term is widely used) strategies have been widely adopted by educators around the world and it is not surprising that it has also become a buzz word amongst medical educators. E-learning, also called web-based learning, online learning, distributed learning, computer-assisted instruction and internet-based learning, can be defined as the delivery of training material via information and communication technology (ICT), including the internet, CD-ROM, DVD, smartphones and other media, both inside and outside of the classroom (Frehywot et al., 2013; Ruggeri, Farrington, & Brayne, 2013; Ruiz, 2006).

Pure e-learning refers to the use of e-learning materials only, without any face-to-face classroom methods. Although it seems to be synonymous to completely online learning, the difference between puree-learning and fully-online learning refers to the delivery platform. While online learning relies on a web-based delivery platform and requires internet access, pure e-learning can take place without internet access, for example, by using DVDs to deliver the educational content. Blended learning combines e-learning components with other traditional face-to-face, lecture-based learning in and outside of the classroom (Frehywot et al., 2013).

Advantages of e-learning include improved accessibility of educational materials at a time convenient to the learner; individualised or personalised learning, where the learner decides on the amount, pace and place, which allows personal tailoring of the learning experience and meeting of individual learner objectives; lower training costs; collaborative learning environments; the ability to track learner activity; and provision of consistent and standardized training to a larger number of learners all over the world (Clark, 2002; Ruggeri et al., 2013; Ruiz, 2006; Welsh, Wanberg, Brown, & Simmering, 2003).

Disadvantages of e-learning include costs related to the technologies as well as the staff needed, although most costs seem to occur at the development stage of e-learning courses; and social isolation or the lack of interaction among the participants of the e-learning activity or course. The lack of face-to-face networking and peer interaction can be seen as a huge drawback, making e-learning less attractive and less useful. Poor instructional design, technical problems and de-individualised instruction can also be regarded as disadvantages of e-learning (Cook, 2007; Welsh et al., 2003).

E-learning is a complex intervention with multiple components and dimensions that interact, in a linear or non-linear way, in a specific context (Wong, Greenhalgh, & Pawson, 2010). In the literature, different components, levels, dimensions and categories of e-learning have been described.

Ruiz et al. (2006) describe the components of e-learning as being i) the development of content ii) management of the content and iii) delivery of the content. Regarding development of content, digital learning objects, i.e. “any grouping of digital materials structured in a meaningful way and tied to an educational objective”, are the fundamental components of lessons, modules and curricula; created with the help of instructional design and pedagogical principles. Examples of digital learning objects include tutorials, case-based learning scenarios, hypermedia, simulations and game-based learning modules (Ruiz, 2006).

Management of content encompasses the administrative functions (storing, indexing, cataloguing) needed to make content available to learners through portals, repositories, digital libraries, learning management systems, search engines and e-Portfolios (Ruiz, 2006).

Content can be delivered in a synchronous or asynchronous way. Synchronous delivery of content refers to real time, instructor-led learning, where all learners receive information simultaneously and communicate and interact directly with other learners by logging onto platforms like virtual classrooms (e.g. teleconferencing, internet chat forums, instant messaging). Asynchronous delivery of content occurs when the transmission and receipt of information occurs at different time points. It can include pre-recorded presentations or podcasts, PowerPoint slides, or even more sophisticated applications like simulations. While asynchronous communication is done by means of email, online bulletin boards, listservs, newsgroups, wikis or weblogs, it enables learners to participate in the learning activity at any time of day, from any desired location (Ruiz, 2006; Welsh et al., 2003).

Ruggeri et al. (2013) describe the different dimensions and attributes of e-learning programmes under four headings: Synchronicity (asynchronous vs synchronous), location (same place vs distributed), independence (individual vs collaborative) and mode (electronic-only vs blended) (Ruggeri et al., 2013).

Cook (2005) describes four levels of instructional design in computer-based learning: Medium, which refers to the mode of delivery of instruction (e.g. textbook, face to face, computer-based, television); configuration, referring to the “big picture” differences within a given media format (e.g. computer: web-based discussion board, web-based or CD-ROM-based tutorial vs face to face: small-group discussion, lecture); instructional method, referring to teaching techniques that support learning processes (e.g. learning activities, self-assessment questions, clinical cases, simulations, group discussions); and presentation, referring to elements of the given medium that enhance the intervention, but do not qualify as instructional methods (e.g. hyperlinks, multimedia, font simulation fidelity etc.) Cook (2005).

We developed a system-based logic model of e-learning of EBHC, based on a template from Rohwer et al. (2016) depicting the different components, contextual factors and interactions that are needed to achieve the desired outcomes (Figure 2). In an effort to teaseout the characteristics and dimensions of e-learning interventions in order to be combined meaningfully, we have only listed the broad categories that we thought of as being most important.

OPEN IN VIEWER

Figure 2

Systems-based logic model of e-learning of EBHC

1.3.1 How e-learning may work

A number of systematic reviews have been conducted in various fields of medical and health science education, examining the effect of different types of e-learning on knowledge, skills and behaviour.

A systematic review conducted by the WHO evaluated the effectiveness of e-learning for undergraduate health professions education (Al-Shorbaji, Atun, Car, Majeed, & Wheeler, 2015). They included 209 studies and concluded that e-learning is similar to traditional face-to-face learning in terms of knowledge and skill acquisition. The majority of included studies were judged to be at high risk of bias.

Cook et al. (2008) conducted a systematic review to determine the effectiveness of internet-based learning for health professionals compared to no intervention or non-internet interventions. The authors pooled data in a random-effects meta-analysis and found significant results favoring internet-based learning for knowledge, skills, as well as behaviour and patient effects when compared to no learning. But when internet-based learning was compared to alternative methods of learning, there was no difference in results between groups. In a subsequent analysis on instructional design variations in internet-based learning, the authors concluded that interactivity, practice exercises, repetition and feedback appeared to be associated with improved learning outcomes (Cook et al., 2010). A review by Booth and colleagues (2009) found that presentation and design, flexibility, peer communication, support and knowledge validation were effective e-learning techniques for enhancing the learning experience of students.

Liu et al. (2016) conducted a systematic review on the effectiveness of blended learning compared to no or other learning amongst health professionals. They included 56 studies and results favored blended learning when compared to no learning and to other learning. However, authors caution that high levels of heterogeneity should be taken into account when interpreting results.

Another recent systematic review assessed the effectiveness of internet-based e-learning on health care professional behaviour and patient outcomes (Sinclair, Kable, Levett-Jones, & Booth, 2016). The authors found 7 studies that assessed behaviour of healthcare professionals and no studies that reported on patient outcomes. They were unable to pool results in meta-analysis due to substantial heterogeneity between studies. Although results from individual studies showed that e-learning was just as effective as face-to-face learning and better than no learning, authors concluded that there was currently insufficient evidence to answer their question.

Other studies have examined how e-learning works, focusing more on the essential components for successful learning, as well as certain contextual factors that influence learning. A realist review by Wong et al. (2010), looking at “what works, for whom and in what circumstances” when considering internet-based medical education, concluded that online courses need to engage the learners to use the technology, which is more likely to happen when the technology is perceived as being useful and easy to use; and that interactivity should be a key characteristic of an online course, since learners want to be able to enter a dialogue with tutors and peers (Wong et al., 2010).

E-learning technologies afford a new learner paradigm based on the adult learning theory, explaining that adults learn by relating new information to past experiences; tailoring learning to their unique needs (open learning); and applying learning in practice, resulting in more effective and efficient learning experiences. The attributes of accessibility and convenience (distributed learning) personalise the learning, because students decide when and where they are receptive to learning. Learning becomes an individual experience where adults learn because they want to learn – not because they are told to learn. This shift from “expert-led teaching to user-lead learning” results from intrinsic motivation and offers a much stronger learning stimulus. Some evidence suggests that learning by means of e-learning is more efficient and that learners gain knowledge faster, which in turn translates back to improved motivation and enhanced learning. (Clark, 2002; Dabbagh, 2005; Ruggeri et al., 2013; Ruiz, 2006).

When considering these attributes and pedagogical underpinnings of e-learning, one can argue that these fit perfectly within the EBHC paradigm. Enhanced learning occurs through internal motivation, rather than external drivers and requires acknowledgement of shortcomings (knowledge gaps) and adoption of a reflective approach towards one's own practice. This alignment between attributes of e-learning and EBHC foci suggests that making use of e-learning in this context could be of value.

The pathway from EBHC e-learning activities to the desired outcomes, i.e. increased knowledge, skill, attitude and behaviour, as well as health outcomes is depicted in Figure 3, a process-oriented logic model, based on the template by Rohwer et al. (2016).

OPEN IN VIEWER

Figure 3

Process-oriented logic model showing the pathway from EBHC learning to desired outcomes

It is important to note that this pathway does not take place in a vacuum, but that ultimately, improved health care delivery and improved health outcomes do not only rely on evidence-based practice, but are influenced by a variety of other factors within the healthcare context e.g. socio-cultural, socio-economic, epidemiological, legal, ethical and political factors on a national and international level (See Figure 2).

3.1.1 Types of studies

We followed the guidance of the Effective Practice and Organisation of Care (EPOC) review group of Cochrane. We included randomised controlled trials (RCTs), cluster randomised controlled trials and non-randomised controlled trials (non-RCTs) in the review. In addition, we considered controlled before-after studies (CBAs) with at least two intervention and two control sites; and interrupted time series (ITS) with a clearly defined point in time when the intervention occurred with at least three data points before and three data points after the intervention.

In addition, we included studies supporting the included intervention studies that yielded information on the process of implementation. These studies were either quantitative (e.g. process evaluations, quantitative interview studies) or qualitative in nature (e.g. focus groups, qualitative interview studies). The information gathered was either reported within the above included intervention studies, or cited within these and published separately. We did not include these supporting studies in the data synthesis, but rather reported relevant results separately.

We excluded studies without a comparison group.

3.1.2 Types of participants

We included all healthcare professionals, including doctors, dentists, nurses, occupational therapists, physiotherapists, dieticians, audiologists, mental health professionals, psychologists, counsellors, social workers at undergraduate, postgraduate (e.g. residents) or continuing medical education (CME) level; working in primary, secondary or tertiary environments.

3.1.3 Types of interventions

3.1.4 Types of outcome measures

3.2.1 Electronic searches

We developed a comprehensive search strategy consisting of relevant terms (Table 9.1) and searched electronic databases (MEDLINE via PubMed, EMBASE via Ovid, ERIC, CINAHL, CENTRAL, SCOPUS, Best Evidence Medical Education (BEME), Web of Knowledge, PsycInfo) and dissertation databases (ProQuest) up to 24 May 2016 for relevant studies. We did not apply any language restrictions.

3.2.2 Searching other resources

We complemented our search with a thorough examination of reference lists of identified studies and contacted experts in the field to identify any on-going or unpublished studies. We also searched trial registries (ICTRP) for on-going studies (24 May 2016).

3.3.1 Selection of studies

Two authors (AR and either TY, ER or NVM) independently screened the search outputs and abstracts for relevant studies. We retrieved full texts of studies with seemingly relevant abstracts and two authors (AR and TY, ER or NVM) independently assessed them for eligibility using the pre-specified inclusion criteria. We resolved discrepancies through discussions and consultations with a third author before classifying studies as included, excluded, or on-going.

3.3.2 Data extraction and management

Two authors (AR and either NVM, TY or ER) extracted data from relevant studies independently and in duplicate using piloted, electronic data extraction forms. Discrepancies were resolved through discussions and consultations with a third author. We contacted authors in case of missing data.

We extracted data related to study design (type of study, duration of study, country), participants (number of participants, type of health care professionals, level of education), interventions (learning theory, educational content, duration, intensity, dose and timing of intervention, delivery of interventions), outcomes (primary and secondary outcomes, measurement details and time point at which outcomes were measured), results (measures of effect with 95% confidence intervals, qualitative data) and educational context (setting, learner context, institutional context and socio-economic context). We considered the term “learning theory” in a broad way, to describe a body of implicit or explicit ideas about how learning works (May et al., 2007; Wells, Williams, Treweek, Coyle, & Taylor, 2012). We extracted theories, models or learning approaches, as defined by study authors.

We broadly categorised interventions and controls into pure e-learning, blended learning, face-to-face learning or no learning. In addition, we indicated whether interventions were single or multi-faceted interventions, whether EBHC learning was stand-alone or integrated, and for all e-learning interventions, whether they required individual or collaborative learning.

In addition, we compiled a matrix of the intervention components related to the included interventions and comparisons in order to compare the components across studies. The matrix was informed by components identified from the included studies and grouped according to mode of delivery, for example, e-learning components and face-to-face components.

3.3.3 Assessment of risk of bias in included studies

For each included study, two authors (AR and either NVM, TY, or ER) independently assessed the risk of bias according to the Cochrane Effective Practice and Organisation of Care's (EPOC) suggested risk of bias criteria (EPOC, 2015) for the following domains: selection bias, performance bias, detection bias, attrition bias, contamination (when participants in the comparison group are exposed to the intervention and vice versa), reporting bias, other bias (Table 9.2).

In addition, we assessed the possibility of additional biases for cluster RCTs according to the Cochrane handbook (Higgins & Green, 2008). •

Recruitment bias: We described whether participants were recruited before or after randomization of clusters. We regarded studies as having low risk of recruitment bias if participants were recruited before randomization of clusters; high risk of bias if they were recruited after randomization; and unclear risk of bias if information about the timing of recruitment is unclear.

We resolved discrepancies through discussion and consultation with a third author if needed.

3.3.4 Measures of treatment effect

3.3.5 Unit of analysis issues

In cluster RCTs, where authors had appropriately adjusted results for clustering, we included the adjusted effect estimates and standard errors using the generic inverse-variance method in RevMan. For outcomes where authors had not appropriately adjusted for clustering, we adjusted the data by calculating the ‘effective sample size’ of each intervention group in the cluster RCT. This was done by dividing the original sample size of an intervention group by the design effect (Higgins & Green, 2008). We then included the adjusted data in the meta-analysis.

For multi-arm studies, which contributed multiple comparisons to a particular meta-analysis, we split the ‘shared’ group (e.g. two intervention groups and one control group) as appropriate to avoid the inclusion of data from the same patient more than once in the same analysis.

3.3.6 Dealing with missing data

We contacted authors in case of missing data. Where authors did not respond, we calculated the means and standard deviations (SD) according to the methods described in the Cochrane Handbook (Higgins & Green, 2008). The mean difference (MD) and SD were then converted to a SMD, standard error (SE) and corresponding 95% confidence intervals (CIs) with RevMan software (RevMan, 2011).

For continuous outcomes, we only included participants for whom outcomes were measured, i.e., per-protocol analysis. For rates of attrition of learners, we used intention to treat analysis, i.e., we included all participants that were randomised to a specific group as the denominator in the analysis.

3.3.7 Assessment of heterogeneity

We assessed heterogeneity by describing variability among studies regarding participants, interventions and outcomes (educational heterogeneity), as well as variability in study design and risk of bias (methodological heterogeneity).

Educational heterogeneity was explored by clearly documenting the characteristics of participants; all components of the intervention relating to intervention design and delivery; as well as outcomes and measurement of outcomes in table format. In addition, the educational context (setting, learner context, institutional context, socio-economic context), in which the intervention was delivered, was explored and reported. See Figure 2.

Methodological heterogeneity was explored by clearly documenting study design as well as risk of bias for each study.

We assessed statistical heterogeneity by looking at the I² statistic as well as the Chi² test statistic. We considered an I² value of more than 30%, and a p-value of less than 0.10 for the Chi² test for heterogeneity to be an indicator of important heterogeneity. In case of considerable heterogeneity, we performed random-effects meta-analysis instead of fixed-effect meta-analysis or reported outcomes in tabular or narrative format.

3.3.8 Assessment of reporting biases

We did not assess reporting biases with funnel plots, since we had less than 10 included studies per outcome.

3.3.9 Data synthesis

Due to important heterogeneity related to the participants, interventions and outcome measurements, we used random-effects meta-analysis to pool results using RevMan. For continuous outcomes, we converted means and standard deviations reported at follow-up to SMD and SEs, and reported the pooled SMD with 95%CIs. For studies where outcomes were reported as mean change scores, we calculated follow-up means from baseline means and used baseline SDs for follow-up SDs before converting to SMDs (Higgins & Green, 2008). For dichotomous outcomes, we reported the pooled risk ratio (RR) with 95% CIs. Studies were grouped under six main comparisons: 1.

Pure e-learning vs no learning

3.3.10 Subgroup analysis and investigation of heterogeneity

We performed the following subgroup analysis on primary outcomes: 1.

Study design: Non-RCTs vs RCTs

3.3.11 Sensitivity analysis

We did not perform sensitivity analysis on primary outcomes, as the number of included studies per outcome was very small.

4.1.1 Results of the search

Our search of the databases yielded 12980 citations (24 May 2016) and we identified three further citations from screening references of included studies and experts. After removal of duplicates, we screened 6175 titles and abstracts, of which we identified 55 as being potentially eligible. We assessed eligibility of these by obtaining and screening the full texts of the studies. We included 24 studies (Table 9.3), classified one study as ongoing (Table 9.4) and excluded 30 studies (Table 9.5) (Figure 4).

OPEN IN VIEWER

Figure 4

Flow diagram of included studies

4.1.2 Included studies

We included 24 studies with a total of 3825 participants in the review. Details of individual studies can be found in the table of characteristics of included studies (Table 9.3). Table 9.6 summarises the characteristics of included studies.

4.1.3 Ongoing studies

We identified one study (Schneider et al., 2014) that is still on-going (Table 9.4).

4.1.4 Excluded studies

From those studies subjected to full-text screening we excluded 30 studies. Reasons for exclusion are listed in the table of excluded studies (Table 9.5).

4.2.1 Selection bias

In 17 RCTs the allocation sequence was adequately generated (Bergold et al., 2013; Bradley et al., 2005; Brettle & Raynor, 2013; Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 2012; Laibhen-Parkes, 2014; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014). Four non-RCTs did not make use of randomisation (Fernandez et al., 2014; Ilic et al., 2013; Ramos-Morcillo et al., 2015; Schilling et al., 2006) and three studies did not describe how the allocation sequence was generated so they were judged as having unclear risk of selection bias (Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 2010). Allocation concealment was adequate in eight RCTs (Bergold et al., 2013; Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 2012), not done in the four non-RCTs (Fernandez et al., 2014; Ilic et al., 2013; Ramos-Morcillo et al., 2015; Schilling et al., 2006) and not described and thus judged as unclear in 12 studies (Brettle & Raynor, 2013; Brouwers et al., 2011; Forsetlund et al., 2003; Hadley et al., 2010; Kamin et al., 2001; Laibhen-Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014).

Baseline outcome measurements were similar between groups in 10 trials (Bergold et al., 2013; Brettle & Raynor, 2013; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 2012; Ramos-Morcillo et al., 2015; Welch, Van Lunen, & Hankemeier, 2014) and not adequately reported in 13 studies (Bradley et al., 2005; Brouwers et al., 2011; Fernandez et al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 2015; Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 2010; Saunders et al., 2016; Schilling et al., 2006). In one study (Laibhen-Parkes, 2014) there was a significant difference in EBHC attitudes between groups at baseline.

Baseline characteristics were similar between groups in six studies (Bradley et al., 2005; Kok et al., 2013; Kulier et al., 2012; Laibhen-Parkes, 2014; Ramos-Morcillo et al., 2015; Schilling et al., 2006), not similar in four studies (Brouwers et al., 2011; Davis et al., 2007; Forsetlund et al., 2003; Saunders et al., 2016), not measured in three studies (Ilic et al., 2013; Ilic et al., 2015; Kulier et al., 2009) and unclear in 11 studies (Bergold et al., 2013; Brettle & Raynor, 2013; Davis et al., 2008; Dizon et al., 2014; Fernandez et al., 2014; Hadley et al., 2010; Horiuchi et al., 2009; Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 2010; Welch, Van Lunen, & Hankemeier, 2014).

4.2.2 Attrition bias

Fourteen studies (Brouwers et al., 2011; Davis et al., 2008; Dizon et al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Laibhen-Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014) were judged as having high risk of attrition bias, due to significant loss to follow up. Five studies (Fernandez et al., 2014; Kamin et al., 2001; Kulier et al., 2012;Ramos-Morcillo et al., 2015; Schilling et al., 2006) did not adequately describe the flow of all participants and were judged as having unclear risk of attrition bias, whereas five studies (Bergold et al., 2013; Bradley et al., 2005; Brettle & Raynor, 2013; Davis et al., 2007; Ilic et al., 2013) were judged as having low risk of attrition bias.

4.2.3 Detection bias

Eleven studies (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Forsetlund et al., 2003; Ilic et al., 2015; Kamin et al., 2001; Kok et al., 2013; Laibhen-Parkes, 2014; Macrae et al., 2004; Schilling et al., 2006) adequately blinded outcome assessors and were judged as having low risk of detection bias. We judged three studies (Fernandez et al., 2014; Ilic et al., 2013; McLeod et al., 2010) as having high risk of detection bias, since outcome assessors were not adequately blinded. The remaining 10 studies (Bergold et al., 2013; Brettle & Raynor, 2013; Brouwers et al., 2011; Hadley et al., 2010; Horiuchi et al., 2009; Kulier et al., 2009; Kulier et al., 2012; Ramos-Morcillo et al., 2015; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014) did not adequately report on blinding and were judged as having unclear risk of detection bias.

4.2.4 Contamination

Eight trials (Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010; Ilic et al., 2015; Kulier et al., 2009; Kulier et al., 2012; McLeod et al., 2010) were adequately protected against contamination. We judged two trials as having high risk of contamination (Bergold et al., 2013; Saunders et al., 2016;); in the remaining 14 trials (Bradley et al., 2005; Brettle & Raynor, 2013; Dizon et al., 2014; Fernandez et al., 2014; Forsetlund et al., 2003; Horiuchi et al., 2009; Kamin et al., 2001; Kok et al., 2013; Laibhen-Parkes, 2014; Macrae et al., 2004; Schilling et al., 2006; Welch, Van Lunen, & Hankemeier, 2014) the risk of contamination was unclear.

4.2.5 Reporting biases

We judged one study (Kulier et al., 2012) as having unclear risk of reporting bias, since the authors only reported the baseline measurements for one of the outcomes. The remaining studies all reported on pre-specified outcomes and were judged as having low risk of reporting bias.

4.2.6 Other bias

We judged five studies (Bergold et al., 2013; Fernandez et al., 2014; Kamin et al., 2001; Kulier et al., 2012; Laibhen-Parkes, 2014) as having unclear risk of other bias and the remaining studies as low risk of other bias.

4.2.7 Additional risk of bias in cluster RCTs

4.3.1 Pure e-learning vs no learning

One non-randomised trial (Schilling et al., 2006) and two RCTs (Laibhen-Parkes, 2014; Welch et al., 2014) compared pure e-learning to no learning (Table 12.1). Overall, results favoured pure e-learning interventions.

4.3.2 Blended learning vs no learning

One non-RCT (Ramos-Morcillo et al., 2015) and four RCTs (Bergold et al., 2013; Dizon et al., 2014; Forsetlund et al., 2003; Kok et al., 2013) compared blended learning with no learning (Table 12.2). Overall, results favoured blended learning interventions for EBHC knowledge and skills.

4.3.3 Pure e-learning vs face-to-face learning

SixRCTs (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010; Horiuchi et al., 2009; McLeod et al., 2010) compared pure e-learning to face-to-face learning (Table 12.3). Overall, there was no difference between groups for any of the outcomes.

4.3.4 Blended learning vs face-to-face learning

One non-RCT (Ilic et al., 2013) and four RCTs (Brettle & Raynor, 2013; Ilic et al., 2015; Kulier et al., 2009; Saunders et al., 2016) compared blended learning to face-to-face learning (Table 12.4). Overall, results were similar between groups for EBHC knowledge and skills, but favoured the blended learning approach for EBHC attitude and behaviour.

4.3.5 Blended learningvs pure e-learning

One non-RCT (Fernandez et al., 2014) and two RCTs (Kamin et al., 2001; Kulier et al., 2012) compared pure e-learning to blended learning (Table 12.5). In Fernandez et al. (2014), the pure e-learning intervention (DVD group) was compared to three control groups: the standard distance group (pure e-learning), the computer lab group (blended learning) and the didactic lectures group (blended learning). We included the two blended learning groups for this comparison. Overall, results favoured blended learning.

4.3.6 Pure e-learning vs pure e-learning

One non-RCT (Fernandez et al., 2014) and two RCTs (Brouwers et al., 2011; Macrae et al., 2004) compared pure e-learning to another form of pure e-learning (Table 12.6). Brouwers et al. (2011) compared three intervention groups. The first group received an online tutorial on AGREE II and was granted access to a PDF copy of the AGREE II tool, the second group received the online tutorial plus a practical exercise and feedback, and the third group received the AGREE II manual. For Fernandez et al. (2014), we included the comparison between the DVD group and the standard distance group. In MacRae et al. (2004) the intervention group participated in an online journal club that included an asynchronous discussion list, while the control group only received the articles per email and had access to electronic journals.