Abstract
BACKGROUND
The Problem, Condition or Issue
Improving educational attainment, while reducing social inequality, is one of the fundamental pillars of education policy across the world (Ananiadou & Claro, 2009; Voogt, Knezek, Cox, Knezek, & ten Brummelhuis, 2013). Information and communications technology (ICT) is often provided by schools or national education departments with the intention of improving attainment and addressing this inequality (SQW Ipsos Mori & London Knowledge Lab, 2011) and yet – despite the need to justify and prioritize sparse educational resources – this spending is often not based upon evidence of effectiveness.
This review will focus on the different forms of ICT in education with the aim to identify which forms yield the most educational benefits for students. Policymakers and schools both require evidence to guide them on which ICT to invest in. It is imperative for teachers, education policy makers, and local planners to know whether and how money spent will impact on the learning of pupils who receive it. This is particularly important in the face of multiple marketing efforts from competing manufacturers and developers and increased demand for ICT in schools from parents and students.
The ICT environment is changing rapidly, and provision, use and expectations of ICT are very different now than only 10 years ago. Technological advancements in the past 5 years mean that smart phones and computers are cheaper and therefore easier to access, both in terms of affordability and community access. This is reflected in ownership. For example, in the UK only 3% of children and young people can now be described as ‘non-users’ of the internet (Office for National Statistics, 2013). This is a very different picture from only ten years ago, and means that ICT provision is operating within very different structures from before then. With basic ICT equipment becoming cheaper, there is no longer evidence of a sharp digital divide in some countries, and instead there is talk of a gap in how ICT is used by those acquiring or receiving it (Liabo, Simon, & Nutt, 2013).
Many countries have invested huge amounts of their education budgets into ICT over the last decade (Haelermans & Blank, 2012): in the US $5 billion was spent by schools on ICT in 2004 (Market Data Retrieval, 2004), while in the UK, secondary schools forecast their 2014/15 spending on ICT will be around $480 million in total (BESA, 2013). Increasingly, as ICT equipment becomes more affordable, some schools are purchasing ICT in order to improve students' attainment, for example by providing each student with a tablet or laptop, offering reduced-rate internet subscription or by ICT-immersion programmes, which embed all classrooms with communications technology and computers. Often these programmes are particularly focused on reaching out to pupils who are under-achieving or who are at a social disadvantage that is perceived to reduce their ability to obtain ICT privately (Lee Watson & Watson, 2011; Mouza, 2008) . This is also happening through work of charities, and across the world (Cristia, Ibarraran, Cueto, Santiago, & Severin, 2012; Finn, Kerman, & LeCornec, 2005; Meyers & Andresen, 2000).
Changes in ICT provision and use are not uniform however. International comparison league tables published by the OECD (OECD, 2010) detail the use of ICT equipment. In 2011 on average 92% of 4th grade pupils had access to a computer at school (based upon a sample of 50 OECD member countries participating in the Trends in International Mathematics and Science Study (International Association for the Evaluation of Educational Achievement, 2011). However, for some of the poorer countries in the OECD (based upon GDP per capita (World Bank, 2014b) this figure was much lower: in Morocco only 69% of students had access to a computer, and in the Yemen this figure was just 28%. For many of the richer countries in the OECD at least 99% of pupils had access (e.g., Qatar and Norway).
The pressures on schools to invest comes from multiple sources, and the pitfalls that exist in terms of purchasing/leasing ICT equipment make it harder for schools to invest confidently. BBC Radio 5 live and the BBC's Panorama programme both conducted investigations in 2012 which exposed the tactics ICT vendors use in order to sell/lease ICT equipment (BBC, 2012; Goldberg, 2012) . Meanwhile, parents and students are both increasingly demanding greater ICT usage in schools (Heinrich, 2012; Matthewson, 2014; Selwyn, Potter, & Cramer, 2010). These pressures mean that schools may not make decisions objectively nor based primarily upon evidence of the effectiveness of such approaches. This review will address these challenges, by reviewing the effectiveness of interventions that provide ICT. Furthermore, this review incorporates studies that have researched the perspectives of students and teachers, in order to illuminate what it is like to be given ICT equipment, for use in learning.
The Intervention
In this review we use ICT as a collective term for stationary computers, laptops and tablets, internet-connected or not. Our interest is in initiatives, interventions or programmes which have provided such ICT individually to students, with the intention of improving their educational outcomes. The provision might be confined to the school environment (for example laptops to be used at school only), to pupils' homes (for example stationary computers at home) or be flexible across all environments (for example laptops which are used both at home and at school). Provision of software as part of a wider hardware package will be included. The provision of ICT might be free, or it might be provided at a discount. We will include studies if they have combined such provision with components to enhance or influence use of ICT in teaching and learning.
The aims of ICT use equally varies around several aspects of learning: to find information, to learn core subjects such as maths and literacy, to learn computer skills, to foster self-regulating learning skills, or to engage people who are physically placed outside of the school (Tondeur, Van Braak, & Valcke, 2007; Vanderlinde, Aasaert, & Van Braak, 2014). However, as stated above, in this review we focus on the provision of ICT equipment (stationary computers, laptops and tablets) to students and this must be a component of the intervention in order for an evaluation to be included. Our boundary for the intervention might be seen as strict and exclusive in a multifaceted ICT learning world. Equally, our extension to interventions which combine provision with teaching might be seen as providing too much variability.
Research which considers the relationship between self-acquired ICT at home and academic achievement and engagement (Biagi & Loi, 2013; Vigdor & Ladd, 2010) is outside of the scope, as is provision of software only, since this kind of use depends on existing ownership of hardware.
How the Intervention Might Work
While improved academic achievement will be an aim for most educational establishments, the rationales for ICT provision can be distinguished along four lines. These are interlinked, but it is useful for the purposes of this review to clarify them. First, the provision of ICT is sometimes related to a concern about access to ICT in the student population. The school may find that some of their students do not have home access to ICT and therefore provide ICT so that their teaching and homework assignments can utilise ICT as a teaching and learning tool. Second, ICT provision can be seen as important for embedding ICT within the school's pedagogical approaches, to enhance instruction. Third, and closely related to the second rationale, ICT can be provided to enhance student motivation, by facilitating self-directed learning (Mouza, 2008). Primarily, therefore, ICT provision is usually implemented with the aim of impacting on students' learning activities, thereby enhancing their academic achievement (Haelermans & Blank, 2012; Mouza, 2008; Penuel, 2006). Fourth, ICT provision can be used as a marketing tool to enhance the number of parents and students choosing a school or to boost a school's/school authority's reputation.
In this context ICT becomes part of the environment in which students learn, both individually for each student as a tool to use within the classroom and/or for schoolwork, and collectively as something all students use and engage with as part of their school environment. So while ICT provision gives opportunities for individual use initiated by the student, it also provides the opportunity for collective learning, and teacher-initiated computer-based instruction. For example, most schools now have specific rooms with stationary computers which teachers book in advance for special computer-based sessions. Having the computers follow students, rather than the other way around, has the potential to change how teachers use computers in their teaching, as well as changing how students use them, because they are available all the time, and because all students have them. Computers can therefore be integrated into learning instead of being a separate item, away from the day-to-day classroom environment (Penuel, 2006). In fact, some have argued that this is the only way to revolutionize learning and teaching (Fleischer, 2012).
While this review is primarily interested in the effects from providing students with personal ICT equipment, an important intermediary variable impacting on outcomes from provision is likely to be teachers' ability or willingness to utilize this in their teaching. Furthermore, we might identify studies that have evaluated schemes where the provision of ICT is accompanied by teacher-level interventions. To complement this important variable, we are including studies which have researched ICT provision from the perspective of students and teachers.
Figure 1 below emphasizes the two main types of ICT provision to be included in the review. The first type includes provision only, such as One Laptop Per Child, where schools give all students, or all students within one year group, a tablet or laptop computer. The second type includes provision alongside further teacher-level intervention such as training on how to use ICT in classroom instructions, booster ICT-lessons, or ICT immersion programmes to enhance the entire school ICT infrastructure. Any effects from either model is likely to be mediated by the school's and teachers' capacity for change (Tearle, 2004).
Logic model of ICT provision (adapted from Zief, Lauver, & Maynard 2006, cited in Anderson et al., 2011)
In addition the model aims to reflect that ICT in learning might also influence, and be influenced by, the relation between teacher and learner (Livingstone, 2012). As shown in Figure 1, the provision of ICT is expected to spur on ICT-related activities which may or may not be conducive to the outcomes of interest to this review, namely academic achievement and school engagement. ICT has additional uses outside of formal education and homework tasks. Provision of ICT also increases access to social networking opportunities, web-based or electronic gaming, and it gives students the opportunity to pursue a wide range of interests which may or may not be relevant to the school curriculum. Social networking, gaming and information-seeking are not inherently negative or positive pursuits. Students may engage in these activities in a whole range of ways, finding a good balance, engaging in risky behaviors, expanding their horizons, meeting new friends or engaging in cyber-bullying. Such activities are intermediary outcomes from ICT provision, which are likely to impact on the outcomes of interest to this review, namely academic achievement and school engagement (third box in Figure 1).
All school-based interventions will invariably impact teachers and students in some way, as they are the people inhabiting the environments which are being changed. In terms of ICT provision, students are the recipients of the intervention and outcomes are measured from their behavior and cognitive development. However, children may or may not use the provided equipment as intended by program initiators who are rarely children or young people themselves. The intermediary outcomes listed in Figure 1 relate to how children themselves perceive and use the ICT provided. These activities can be counted, but equally important are children's reasons for this behavior: their underlying perceptions of ICT provision.
It is important to consider Figure 1 in relation to the wider micro structures such as family and individual characteristics, meso structures such as individual school characteristics, and macro structures such as ICT infrastructure and social class (Biagi & Loi, 2013). In terms of provision of ICT to students, its impact on learning within the school environment will be related to the school's capacity for change, in this context “the collective competence of a school to implement ICT in a way that is a lever for instructional change” (Vanderlinde & van Braak, 2010, p. 543). Figure 2 below, illustrates the wider structures in which ICT provision operates.
Conceptual framework
This review will be using the logic model (Figure 1) when considering relevant studies for the review. The conceptual framework (Figure 2) informs our selection of variables for moderator analysis (see page 27) and will help us contextualize the findings of included studies. For example, there is some evidence indicating that social class and gender are both wider variables which will intervene across pathways of impact from ICT provision (Elwick, Liabo, Simon, & Nutt, 2013; Eynon, 2009). The included studies on teachers' and students' perspectives on ICT provision might further help illustrate these contextual influences.
Why it is Important to do the Review
There are many reasons why this review is important, including the costliness of providing ICT interventions, the policy and educational goals tied to ICT interventions and the murky picture relating to the overall effects of these interventions.
Knowing which ICT yield the most educational benefits for students is a challenge for schools. A challenge exacerbated by the fact that they must contend with parents advocating urgently for ICT in schools, pupils requesting ICT services and vendors overwhelming them with skewed advertising data. Knowing whether to invest in particular ICT, and if so, what size of investment, is critical in the context of tight budgets. Schools and educational authorities find themselves in the middle of this dilemma, wrestling to find solutions.
Children and young people aged 4-18 spend a large proportion of their lives in school where they learn and engage in both academic and social enterprises. School is therefore an essential influence on their intellectual and social development. When considering school-based ICT interventions, it is important to include pupils' perspectives for two main reasons. First, according to the United Nation's Convention of the Rights of the Child, children have a right to have their views heard in matters that concern them. Second, since the outcomes sought by ICT enhancement programmes are directly connected to children's outcomes, children are central to the intervention. There might be a popular assumption that ICT provision is only perceived positively by children and young people, but there is the possibility that they may feel situational pressures to use ICT, against their own inclinations. Or, if they do indeed value ICT provision highly, this ought to be an important factor for consideration alongside other outcomes. As a result, understanding and incorporating student perspectives is an integral part of the review in terms of the inclusion of studies capturing these, as well as an important outcome of the overall review. Finally, including children's views relates to the overall aim of considering ICT impact on school engagement, since views studies might further illuminate this particular aspect of ICT interventions.
ICT intervention studies have evaluated whether ownership of ICT impacts educational attainment (Fairlie & London, 2011; Fairlie & Robinson, 2011). These studies examined the impact of ICT on educational attainment by providing students with free laptops. Other intervention studies provided computer skills training along with discounted or free computers and internet access (Finn et al., 2005; Tsikalas, Lee, & Newkirk, 2007). In addition, some studies considered whether the use of internet impacts educational achievement (Jackson et al., 2006), and whether the use of ICT in both teaching and learning impacts achievement (Low & Beverton, 2004). While the findings of these studies vary, rigorous and recent individual studies have suggested that there is reason to question the effect of ICT provision on achievement (Belo, Ferreira, & Telang, 2010; Cristia et al., 2012; Fairlie & London, 2011; Fairlie & Robinson, 2011).
Despite an accumulation of results from individual ICT intervention studies, a lack of clarity remains on the effect of ICT provision. A systematic review from 2006 concluded in a narrative synthesis that one-to-one laptop programs appeared to impact positively on technology use, technology literacy and writing skills (Penuel, 2006). A rapid systematic review of ICT interventions found both that there are mixed results from most studies, and that the overall effect seemed to be negligible (Liabo et al., 2013). Liabo et al (2013) considered intervention studies in which ICT were provided to pupils as well as intervention studies which focused on self-acquired ICT equipment (mainly computers and internet access at home). We have not found any systematic reviews on pupils' views on ICT provision.
A comprehensive systematic review of both ICT intervention studies and studies of young people's ICT views is therefore imperative for schools, educational authorities and all who work and invest in the education sector. This Campbell systematic review will build on the previous rapid review by Liabo et al (2013). It will conduct a more comprehensive literature search and focus on the provision of specific ICT and technology services programs. We expect the results of this review to be of practical use to policy makers, education planners, school heads and governors who make budgeting decisions and draw up strategies for school improvement.
OBJECTIVES
This review has two main objectives. The first objective is to address the question “What is the effectiveness of ICT provision on pupils' school engagement and achievement?” by systematically reviewing, assessing and synthesising the evidence from randomised and quasi-experimental evaluations of such interventions. A sub-objective of this is to consider separately, such provision targeted at socially disadvantaged students. The second objective is to address the question “What are children, young people, and teachers' views and experiences of ICT provision?” by systematically reviewing, assessing and synthesising the evidence from primary research (qualitative studies and surveys) which have addressed this question.
METHODOLOGY
Criteria for Including and Excluding Studies
Types of Participants
This review focuses on children aged 4-18 who are in compulsory education at elementary/primary level or high/secondary level, living in a high income country in Europe, North America or Australasia, as defined by the World Bank (World Bank, 2014a). This is because there is considerable difference in ICT access and availability between high-income countries and those of lower-middle income, and because the policy aims of introducing ICT in schools are likely to vary across very different socio-economic settings. The countries included are therefore: Australia, Austria, Belgium, Canada, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Iceland, Ireland, Italy, Luxembourg, Netherlands, New Zealand, Norway, Poland, Slovak Republic, Slovenia, Spain, Sweden, Switzerland, United Kingdom, and the United States. The defined age span reflects the ages of compulsory schooling in these countries.
In regards to the sub-objective of objective number one, by social disadvantage we refer to students who meet the EU social inclusion indicator of relative poverty (60% or less of the median income 1 ) or those eligible for free or partially subsidised school meals. Also included within our definition of socially disadvantaged will be pupils where multiple factors indicate social disadvantage, most notably living in a poor area with a low household income and with parents without higher education. Pupils characterised by living in foster or residential care will also be considered to be at social disadvantage and defined within this group.
In regards to objective number two we will include studies which have asked teachers' views on ICT provision programmes, as well as those asking students.
This review will not include studies on students who have special educational needs, or who have been excluded from school on the basis of their behaviour. A separate title for assistive technology for people with autistic spectrum disorder is registered in the Campbell Library (Nikopoulos, Nikopoulo-Smymi, & Dillenburger, 2012). ICT provision for people with special educational needs or behaviour problems is likely to serve purposes going beyond those implemented for all students.
Types of interventions
As described earlier in this protocol, this review focuses on interventions which provide students with free or discounted computers, including tablets, to be used at home, at school, or at a community centre. This can be programmes only providing computers, computers in combination with internet connection, or provided along with another learning package such as curriculum innovation or professional development (e.g. teacher training in instructional ICT use). This also includes ICT immersion programmes, which are a whole school approach to technology with an emphasis on ICT across the school environment and for individual use by students in their learning.
Comparisons can be “no intervention,” another type of intervention to improve achievement or school engagement, or comparisons between different kinds of ICT provision interventions at least one of which adhere to the intervention definition provided here.
Excluded will be programmes that have implemented ICT for teachers to influence their teaching, such as interactive whiteboards or online teaching planning, unless these were accompanied by additional ICT provision to students. Also excluded will be instructional programmes, and online learning programmes, such as Mathletics, again unless they were provided alongside additional ICT provision as described above.
In relation to objective 2, studies which have asked children, young people or teachers about their views, experiences or use of ICT in general, and not in connection with a relevant intervention programme (i.e., one that meets the above criteria for inclusion in this review) will be excluded.
Types of study designs
Objective no. 1: To address objective number one (systematic review of the effect of ICT provision), we are interested in quantitative effect sizes derived from study designs where one or more groups receiving the intervention are compared with one or more groups not receiving it.
The method of allocation can be randomisation and quasi-randomisation. Randomisation here refers to when the randomisation was true, for example computer-generated. Quasi-randomisation refers to when the randomisation was by a particular variable, for example date of birth or toss of a coin. We will also include quasi-experimental studies where the group allocation was not random.
For quasi-experimental studies to be included, they will have had to consider the equivalence between groups and, if not found, to have addressed this by use of a statistical measure (for example propensity score matching or use of statistical controls). For quasi-experimental studies, baseline equivalence will be determined by the magnitude of an effect size. However, consideration of equivalence must include at least one of the outcomes of interest to this review and we will only include quasi-experimental studies for which there was equivalence on pre-test for at least one of the primary outcomes listed in this review. Outcomes for which there was no equivalence at pre-test will not be included. Equivalence is also addressed in the critical appraisal tool (selection bias assessment for quasi-experimental studies).
The unit of allocation can be individuals, where students within one school received free or discounted ICT and were compared with those in the same school who did not schools, or by clusters, where schools are allocated and individuals are therefore in a group due to their attending a particular school. The advantage of cluster randomisation is that it controls for contamination between individuals within the same setting.
Examples of included study designs to assess the impact of ICT access and provision interventions are provided in Table 1 below.
Examples of effect study designs
Excluded designs for objective number one are: Before-after studies: comparing results from the intervention period with previous periods in the same setting—for example, looking at in-school variation by year considering academic results or engagement levels before the introduction of ICT provision and after.
Objective no. 2: To address objective number two (systematic review of children and young people's views on ICT provision) we will include: Case study and ethnographic designs facilitating qualitative interviews with students who received an intervention of enhanced ICT provision. This will include one-to-one open and semi-structured interviews and focus groups. Quantitative and qualitative surveys assessing students' satisfaction of receiving an intervention of enhanced ICT provision.
Because focus groups and qualitative interviews are often carried out in the qualitative arm of a trial we envisage that some studies will include a comparison group. However, this is not a criterion for inclusion since this assessment does not concern whether the computers changed pupils' experiences.
Examples of the kinds of study designs included for the assessment of children and young people's experience of and views on ICT enhancement programmes are listed in Table 2 below.
Examples of experience and views designs
Excluded designs for objective number two are: Studies that have not interviewed students or teachers about their views.
Types of outcome measures
Objective no. 1: Outcome measures relevant to objective number one (systematic review of the effect of ICT provision), and its sub-objective (to consider these effects on socially disadvantaged students) are: Student achievement, as measured by literacy, numeracy, or tests on a particular subject, in written or oral tests, or as changes in grade point averages. We will include studies which have employed validated outcomes tools for this purpose, as well as studies which have used routine data based on tests. Student engagement with school, as measured in standardised questionnaires (for example the Young Children's Computer Inventory), attendance patterns, exclusion numbers, or indicators on school enjoyment and participation measured on standardised tools (Libby, 2004). Outcomes corresponding to the intermediary outcomes listed in the logic model (Figure 1, page 6): time spent on homework, time spent on extra-curricular activities using the ICT provided by the programme, ICT knowledge, levels of direct student-teacher contact, amount of cyber bullying or online grooming (where people are approached for sex online), amount of ICT use for independent learning tasks. These will include non-validated measures.
Outcome measures listed will not be used as criteria for including studies.
Excluded outcomes
Students' self-reported achievement levels or test results. Teachers' subjective reports of students' progress and achievement.
Duration of follow-up
We will include measures on outcomes which have been measured one term or more after the introduction of the intervention.
Types of settings
We will include interventions which provide ICT to students at school, home or a community centre, as long as the aim of the intervention, and therefore the outcome measures, relate to the children and young people's educational achievement or engagement as defined above.
Date, language and form of publication
A rapid systematic review (Liabo et al., 2013) searched from year 2000 onwards and the oldest included study identified was published in 2003. Judging from this, and the rapid developments in ICT ownership described in the introduction, we will search for studies published in 2003 onwards. This will capture studies which were conducted in the build-up to the introduction of smart phones, and social networking sites, since iPhone was launched in 2007, and complete public access to Facebook in 2006. This will also include the time period in which the one-laptop-per-child was developed and launched.
We will aim to include non-English literature, but this will depend on resources available. At present, we have resources to include Scandinavian (Danish, Swedish and Norwegian) and German language studies.
No study will be excluded on the basis of its publication form.
Search strategy
The review will be informed by one comprehensive search to identify studies for both objective 1 and objective 2.
Search terms
The following concepts will form the basis of the search strategy:
Intervention: information communication technology (e.g., technology, laptop, computer, mobile, internet, wi-fi, ICT immersion) AND access (e.g., provision, free, discounted). The search will include terms for brand names such as one-laptop-per-child and Microsoft Anywhere Anytime Learning
AND
Population: school-aged children (e.g., young people, pupils, students)
AND
Setting/outcomes: education (e.g., school, learning, training) OR achievement (e.g. grades, test results) OR school engagement (combining setting and outcome)
These concepts will be combined in a search string similar to:
((ICT NEAR TO (access OR support OR provi* OR free)) OR ((information WITHIN 2 communication* WITHIN 2 technolog*) NEAR TO (access OR support OR provi* OR free)) OR (computer* OR laptop* NEAR TO (access OR support OR provi* OR free)) OR (internet NEAR TO (access OR support OR provi* OR free)) OR (wi-fi NEAR TO (access OR support OR provi* OR free)) OR (wifi NEAR TO (access OR support OR provi* OR free)) OR ((wi WITHIN 2 fi) NEAR TO (access OR support OR provi* OR free)) OR (www NEAR TO (access OR support OR provi* OR free)) OR ((world WITHIN 2 wide WITHIN 2 web) NEAR TO (access OR support OR provi* OR free)) OR (telecommunications NEAR TO (access OR support OR provi* OR free)) OR ((information WITHIN 2 technolog*) NEAR TO (access OR support OR provi* OR free)) OR ((technolog* WITHIN 2 integration) NEAR TO (access OR support OR provi* OR free)) OR ((education WITHIN 5 technolog*) NEAR TO (access OR support OR provi* OR free)) OR ((school WITHIN 5 technolog*) NEAR TO (access OR support OR provi* OR free)))) OR (digital NEAR TO (access OR support OR provi* OR free))
AND
((child* OR (young WITHIN 2 person*) OR (young WITHIN 2 pe*) OR boy* OR girl* OR teenage* OR schoolchild* OR youth* OR adolescent* OR juvenile* OR student* OR pupil*)
AND
Educat* OR school* OR classroom* OR achievement OR (education AND (engagement OR enjoy*)) OR (school AND (engagement OR enjoy*))
Corresponding terms used by individual databases will be identified and used so that both free-text and thesaurus searches are carried out.
Search terms associated to each of these concepts will be combined using AND; however, in the event of hits exceeding 5,000 in one database, NEAR, WITHIN or similar Booleans will be used between the terms for setting and terms for intervention. Study design filters will be used if the initial search exceeds 25,000 hits.
Sources
Test searches will be carried out in two of the following databases which have been identified as particularly relevant: ERIC BEI (British Educational Index) ASSIA (Applied Social Sciences Index and Abstracts) LISTA (Library, Information Science & Technology Abstracts) Sociological Abstracts Technology Research Database Child Data Social Care Online Bibliomap SSCI (Social Science Citation Index) Scirus EdITLib OECD Proquest Education Journals PsycInfo Omnifile full text
To boost this strategy's capacity for locating dissertations, theses, and other unpublished research we plan to: Specifically search for grey literature in: Web of Science, Science Direct, Springer link, Education full text, and Dissertation Abstracts International. Reference harvest: All included studies and identified reviews and meta-analyses with a focus on ICT provision in education will be hand searched and authors contacted. In addition, a Google search will be conducted to identify localities where ICT provision has been implemented, and further contact made to ask whether any evaluations were conducted of such initiative. Conduct a second website search: screening websites of ICT charities within OECD countries, searching Contact authors of all relevant papers for further information and asked whether they are aware of any further evaluations, specifically any unpublished work including PhD theses.
Description of Methods Used in Primary Research
The rapid review (Liabo et al., 2013), which this review is updating and expanding, identified eight evaluations of ICT provision programmes. Six of these scored low on methodological quality, because they did not use randomisation or matched comparisons, and tended to have small sample sizes. Judging from those findings we expect to find a relatively large number of small-scale studies, without a comparison group. We also expect to find many studies which have asked students about their grades, rather than using their academic grades before and after the intervention. Both these characteristics are listed in our exclusion criteria. Liabo et al. (2013) found one large randomised-controlled trial and of high methodological quality (Fairlie & Robinson, 2011). This study, which is detailed briefly below, meets the criteria for this review
Fairlie, R. W., & Robinson, J. (2011). The effect of home computers on educational outcomes: Evidence from afield experiment with school children Working Paper #11-14: NET Institute.
Abstract: Are home computers an important input in the educational production function? To address this question, we conduct afield experiment involving the provision of free computers to schoolchildren for home use. Low-income children attending middle and high schools in 15 schools in California were randomly selected to receive free computers and followed over the school year. The results indicate that the experiment substantially increased computer ownership and total computer use among the schoolchildren with no substitution away from use at school or other locations outside the home. We find no evidence that the home computers improved educational outcomes for the treatment group. From detailed administrative data provided by the schools and a follow-up survey, we find no evidence of positive effects on a comprehensive set of outcomes such as grades, test scores, credits, attendance, school enrollment, computer skills, and college aspirations. The estimates also do not indicate that the effects of home computers on educational outcomes are instead negative. Our estimates are precise enough to rule out even modestly-sized positive or negative impacts. The lack of a positive net effect on educational outcomes may be due to displacement from non-educational uses such as for games, social networking, and entertainment. We find evidence that total hours of computer use for games and social networking increases substantially with having a home computer, and increases more than total hours of computer use for schoolwork.
The main criteria for including studies for
Sclater, J., Sicoly, F., Abrami, P. C., & Wade, A. C. (2006). Ubiquitous technology integration in Canadian public schools: Year one study. Canadian Journal of Learning and Technology, 32(1).
The current investigation was an exploration of the first year of a multi-year project designed to provide every Grade 3 to Grade 11 student throughout an English school board in Quebec with a laptop computer. Data were collected from 403 elementary and 270 secondary students from the experimental school board and also from 330 students in the control school board. In addition, questionnaire data were collected from 60 elementary school teachers and 51 secondary school teachers. Finally, interviews were conducted with 72 students and 20 teachers. Potentially the most interesting finding was the difference in achievement scores between the experimental and control boards. Secondary students from the experimental board had higher scores on the CAT-3 reading test and indicated making six times more frequent use of computer technology in their English classes, suggesting a possible treatment effect. In contrast, math scores were higher at the control board where neither board indicated high levels of computer use. Nevertheless, these findings must be interpreted with some caution until the threats to validity of selection bias are more clearly overcome.
Criteria for Determination of Independent findings
It is likely that we will find some studies which have been published across several reports. If this occurs, we will assess which is the most relevant and rigorous report and start data extraction from this. However, the other reports will be linked and scrutinised for additional or contradictory data. Authors will be contacted if clarification is needed. Reviews or combined study reports will be unpacked and individual studies used.
This review will focus on two main outcomes: school engagement and achievement. Achievement is defined here by results from tests, and is therefore a fairly straightforward outcome to measure. Measures of engagement might vary considerably. For example, some studies might have used measures of behaviour rather than measures of attitudes to school or learning. Engagement as a construct therefore relates to three main outcome types: attendance measures (rates of attendance, truancy, exclusions), measures on pupils' relationship to school (attitudes, attachment, enjoyment), and measures on pupils' behaviour. These three constructs will not be combined, nor will engagement and achievement as overall constructs.
This means that we will not combine behaviour outcomes with psycho-social attitudes outcomes. For example, if one study has used the Strength and Difficulties Questionnaire, which is a behaviour and well-being questionnaire (Goodman, 2001), or a self-esteem scale, outcomes from these will not be combined with measures on attitudes to school, sense of school membership or future expectations scales (e.g. Furnham & Rawles, 1996; Goodenow, 1993), nor with attendance outcomes.
However, if most studies employ several standardised measures on one of these constructs on engagement, we will combine these into one overall measure of effect per study. The same goes for measures of attendance. If most of the included studies have only used one measure for assessing a construct, studies using several measures will be assessed, and only the most similar across the studies will be used. Achievement measures will also be pooled into one overall effect size and effect sizes by subjects (mainly maths, reading and writing, but also other subjects such as history and science, if available). We will convert all effect sizes into the same metric as long as this is defensible according to Campbell methodological standards (The Campbell Collaboration, 2014).
Details of Study Coding Categories
Selection of studies
All search hits will be imported into EPPI-Reviewer 4, which is a software developed to support all stages of a systematic review.
2
Duplicates will be identified and removed, before screening begins. Titles and abstracts will be screened on a hierarchy of codes: Not about the provision of ICT to education/community/individuals Not about provision to students in compulsory primary or secondary schooling Not a research evaluation with individual data Lacks focus on primary outcomes/students' views on ICT provision Out on country or year Include for second opinion Include for full-text screening
The screening tool above will be tested within the team, inter-rater reliability will be measured, and the tool adjusted accordingly. There will be further double-screening of at least 10% of the studies but we will go beyond this percentage if required by the inter-rater reliability and using the Kappa test aiming for a score of 0.75. This may result in all search hits being screened by two reviewers or more if we do not reach this level.
Full texts will be retrieved for all items that appear to meet the selection criteria on basis of the information in the title and abstract. These items will be re-examined against the criteria outlined in this protocol, using all the information in the full study report. Full text screening will be conducted by two reviewers per study.
Data extraction and critical appraisal
Full texts of the included studies will be assessed by two reviewers for each study, on four data extraction and critical appraisal tools, draft versions of which are provided in Appendix 1 and 2. These tools will record all important information on the study's context (authors, country, intervention components, population, outcomes), and assess the quality and relevance of the study design.
Objective no. 1: The methodological quality of quantitative evaluations (to address objective number one) will be assessed on a tool similar to those developed by the Cochrane Methods group and adapted for non-randomised studies (J. Higgins & Green, 2011). 3 The domains are listed below, with full details of the tool included in Appendix 2. The questions on selection bias will address aspects of the sampling (selection of and similarities between the groups). Performance bias will always be a particular concern in educational studies because of the problems of blinding an educational intervention; however, we will address the extent to which the groups differed in the education they received (apart from the intervention under investigation). Attrition bias relates to the loss of participants and availability of outcome data. Detection bias relates to how the outcomes were assessed, and, where applicable, whether the assessors were blinded to the identity of the material they assessed (awareness of its relationship to allocation groups, anonymous assessment). Reporting bias will relate to the extent to which the authors' conclusions match their reported findings, and whether there is reason to trust the authors' and/or the funders' intent for the study.
Objective no. 2: Qualitative studies (to address objective number two) will be assessed using a modified version of the CASP tool
Satisfaction surveys will be assessed on the appropriateness of the design, measurement and analysis of data. Particular emphasis will be placed on the sampling, representativeness, rigorousness of measuring tools, relevance to the review focus and conflict of interest (funding source).
All included studies will be critically appraised by two reviewers. Differences will be discussed and consolidated in collaboration with a third reviewer. Due to resource limitations the main data extraction will be conducted by one reviewer, but quality appraised by someone else in the review team.
Statistical Procedures and Conventions
The statistical procedures for this review relate to objective number one (effectiveness review) and there are two main outcome constructs under investigation: first achievement, second school engagement. As described earlier, these two main outcomes will be analysed separately since they are qualitatively different. Objective number one also includes a subgroup analysis of effects on socially disadvantaged students.
Calculating effect sizes
Effect sizes will be calculated for each study, using EPPI-Reviewer 4 or other web-based resources, such as the Campbell Collaboration's effect size calculator. 4 All effect sizes will be coded so that positive effect sizes represent positive outcomes.
We expect the majority of outcomes in the included studies to be continuous. For continuous outcomes we will calculate the standardised mean difference following the procedures outlined in Lipsey and Wilson (2001).
There might also be studies which have used dichotomous outcomes, for example passed versus failed exams or school exclusions. For dichotomous outcomes we will calculate the odds ratios (drawn from relative frequencies and proportions of events in the allocation groups).
Unit of analysis issues and cluster adjustments
Unit of analysis errors occur when interventions are allocated at a different level (e.g. cluster) than the unit of analysis (e.g. individual). Since clustering has the effect of narrowing the confidence intervals from the true confidence intervals, the standard errors cannot be calculated using the usual formula (Hedges, 2007). The standard errors of the effect sizes have to take into account the intra-cluster correlation in the outcome variable, otherwise the precision of the estimates will be overstated. To correct for variation associated with cluster-level assignment, the unit of assignment to treatment and comparison groups will be coded for all studies, and appropriate adjustments made to effect sizes using the methods described in the Cochrane Handbook for Systematic Reviews of Interventions (J. Higgins & Green, 2011).
Corrections for small sample sizes
To correct for small sample size, all SMD effect sizes will be converted to Hedges' g, a standardised mean difference with a small sample size bias correction factor (Hedges & Olkin, 1985).
Conversions into a common metric
For some outcomes it is possible that we will encounter the use of a continuous outcome by some authors, and a dichotomous outcome by others. Therefore, for each outcome category, we will determine the number of coded effect sizes in each of the different metrics. Where more than one type occurs in a given outcome category, we will transform the effect size metric with the smaller proportion into the metric with the larger proportion using the Cox transform as shown by Sánchez-Meca, Marin-Martinez, and Chácon-Moscoso (2003). This will allow all the effect sizes for that outcome category to be analysed together. In the event that we do not have consistency across our data (i.e., effect sizes based on either all raw data or all log-transformed data), Higgins, White, and Anzures-Cabrera (2008) will be consulted for guidance on data transformation. We will conduct sensitivity analysis for any transformations made.
Synthesising effect sizes
If there are enough data from rigorous studies, we will conduct a meta-analysis of the outcomes for which we have relevant data. By ‘enough’ data we mean at least two independent studies which provide effect sizes on the same outcome constructs, with similar design and comparison condition. The analysis will be conducted in EPPI-Reviewer 4. The decisions to trust, and therefore use, pooled effect sizes will be informed by the test for heterogeneity described later.
Effect sizes will be pooled separately for RCTs and non-RCTs, and presented separately. However, if we identify more than one RCT we will also consider pooling across these two design categories (e.g., where RCT and non-RCT studies are found to be homogenous), and if we do so our rationale will be explicit. This will allow for greater statistical power. The synthesis will also be conducted separately for studies comparing two ICT provision programmes, and studies comparing ICT with no treatment.
Data synthesis of primary outcomes will be carried out using random effects statistical models, unless a compelling case arises for fixed effect analysis. To account for varying sample sizes and precision across the effect sizes, we will weigh each effect size by its standard error (inverse variance weight) as in Hedges' adjusted g (Deeks, Altman, & Bradburn, 2000). This means that larger n studies will be given more weight in the analysis.
Effect sizes within each outcome construct (achievement, attendance, behaviour and relationship to school) will be pooled into an overall estimate of effect. Forest plots will be used to display the estimated effect sizes from each study along with their 95% confidence intervals. In the event that there is insufficient similarity to statistically combine the study results, forest plots will be presented showing only the point estimate and error measurements for each study.
In relation to our sub-objective of analysing results in relation to socially disadvantaged students, this will include a separate analysis focusing on the findings of studies which evaluated interventions targeted at this group, following the same steps as outlined here for all studies.
Publication bias
We will address publication bias in two ways. Firstly, we will attempt to minimise the possibility of publication bias by conducting an extensive search of the published and unpublished literature. Secondly, the presence of publication bias will be evaluated empirically. Visual analysis of funnel plots, the ‘trim and fill’ method (Duval & Tweedie, 2000) and/or Egger's regression based-test (Egger, Davey Smith, Schneider, & Minder, 1997) will be used to assess the possibility of publication bias in the analytic sample and its potential impact on the findings of the review.
Missing data
In the case of missing data, where a question in the tool cannot be answered from the information in the obtained report, we will contact authors if the missing data will impede on our analysis. We will also search for additional reports of the same study, since articles published in peer reviewed journals often contain less information than the original technical report to the funder, for example. If we cannot obtain the missing data through these sources, we will employ the approach of applying replacement values, complemented by explicit reporting of the methods used, and discussion on the potential impact of the missing data on the findings (Higgins & Green, 2011).
Heterogeneity and planned sensitivity analyses
Consistency across findings will be visually examined using the forest plots. If confidence intervals for the results of individual studies have poor overlap, this may indicate the presence of statistical heterogeneity. Statistical tests (Q, I 2, and τ2) will also be used to assess whether any observed variability in effect-size estimates are attributable to true systematic variation rather than sampling error (Deeks, Altman, & Bradburn, 1997; Higgins & Thompson, 2002). Tests for heterogeneity will be carried out using random effects models.
Sensitivity analyses will be conducted to test the robustness of the results of the data synthesis and offer possible explanations for the differences between studies when interpreting the results. Where possible, we will examine whether the pooled estimates of effect are sensitive to: (a) the study design and methodological quality of studies, (b) outliers, (c) the specific statistical procedures and methods for computing each effect size, (d) our method of analysis - in particular, decisions relating to transformation between effect size metrics, the way outlier effect sizes and sample sizes were handled, and missing data imputations, (e) the degree of missing or incomplete data, (f) the way outcomes were measured in the primary studies, and (g) the timing at which outcomes were measured.
If significant heterogeneity is evident, further analyses will explore possible sources of that heterogeneity (see text on moderator analysis below).
Moderator analysis
We will code moderator variables which relate to the characteristics of the young people included in the studies (baseline achievement and engagement levels, social class, gender), the schools and neighbourhoods included (crime levels, poverty rates, employment rates), the intervention (setting, intensity, duration, level of integration with the learning approaches, level of technical support, theory of change), and study design (randomisation, attrition). These variables include the main outcomes of interest to this review, and correspond to the conceptual framework informing this study (Figure 2 on page 8).
Having specified a priori what might cause the results to differ, we plan to conduct moderator analyses if there are sufficient data. The analysis of these moderating variables will aim to ascertain whether these variables are associated with larger or smaller treatment effects. It is unlikely that we will have the minimum requirement of ten studies of sufficient quality for each moderator variable that would allow the use of meta-regression models (Borenstein, Hedges, Higgins, & Rothstein, 2009). In this event, we will use an analogue to the ANOVA analysis (univariate) approach, as described in Lipsey and Wilson (2001). Power calculations will be conducted for these analyses (Hedges & Pigott, 2004).
Alternative approach to synthesising quantitative data
We may only find a very small number of studies. If their measured outcomes cannot be pooled because they are within different constructs (for example, one study on achievement, and another one on self-esteem) then a narrative analysis will be conducted. However, use of a narrative analysis will mean it will not be possible to arrive at a conclusion as to the effect of enhanced ICT provision programmes. The narrative analysis will focus on the theories of change and whether there appeared to be a difference in results according to the programme characteristics and rationales. This will aim to inform theory building and will not be related to programme impact.
Treatment of research on young people's views and experiences
The screening of qualitative ‘views’ studies will be conducted in conjunction with the screening for impact evaluations, and will therefore follow the same procedure in terms of double screening and assessment. As stated above, studies will be assessed on the appropriateness of the study design, the ‘fit’ between the study question and the study methods, the amount of contextual information provided and the extent to which the researchers linked findings to the context of the experiences. The assessment of satisfaction surveys will reflect similar concerns, as well as an interrogation of the sampling strategy and representativeness of the sample, and the validity of the data collection tool.
The synthesis of views studies will be conducted in three stages, following the protocol of a thematic synthesis (Thomas & Harden, 2008).
Data extraction for the qualitative synthesis will follow different principles compared to the extraction of data from the impact evaluations. Qualitative research will be examined using inductive method where data is read and coding categories emerge from this reading and rereading of results. As explained by Thomas and Harden (2008), this entails in a first step the formulation of descriptive data codes; that is, reported findings in the included studies (e.g., direct quotes or authors' explanation of participants responses) will be assigned a code that describes the phenomenon as it is reported in the study. In a second step, these descriptive codes will then be grouped and combined into descriptive themes. This process is still at the level of the individual study and therefore remains mainly descriptive. In a third and last step, the descriptive themes identified in each of the included studies will then be juxtaposed with each other in order to configure their overlap and relation. This process of configuration is aimed to produce analytical themes, which present the output of the qualitative synthesis.
The overall aim of this thematic synthesis will be to identify patterns and also contextual information about how children, young people and teachers have experienced the intervention at school, in terms of their usage in learning and teaching, and in terms of other socialising aspects including school engagement. Contextual information will include young people's experiences with using ICT, whether any technical problems were experienced and how they were solved (or not), and whether the wider environment impacted on their experiences of having more ICT access, for example whether having a laptop in their school bag increased their risk of crime. The logic model in Figure 1 will inform the analysis so that it addresses the key aspects of the theory of change.
Satisfaction surveys will be examined by considering key findings on positive and negative aspects of ICT provision programmes. Again, each finding will be linked to the certainty of the evidence based on an assessment of consistency and quality. The two analyses will be merged, convergence and divergence within the two datasets will be explored and further analysed. This will function as a type of triangulation, whereby young people's qualitative accounts of ICT provision or enhancement is compared and contrasted with survey data from larger samples.
Second, data will be examined using categories emerging from the analysis of the impact studies. At this stage, interventions in the ‘perspectives’ studies (qualitative studies and surveys) will be compared with those in the quantitative evaluations. Studies will be grouped by interventions, and effect sizes and views compared and congruence or divergence of findings will be considered and discussed. Drawing on Harden and Thomas (2005), this will be done using a matrix that supports a constant comparative analysis of the themes identified in stage 1 of the review of objective 2, and the findings from the review of objective 1. The analysis will identify areas of convergence and divergence.
In addition to the analysis of primary data, the theory of change of each study will be examined to assess the role of pedagogy in the intervention design. If deemed appropriate, findings on effects and views will be considered alongside this, to investigate whether there appears to be any relationship between the theory of change and the findings on achievement and engagement.
Footnotes
REVIEW AUTHORS
|
|
| Title: Senior Research Fellow |
| Affiliation: University of Exeter |
| Medical School |
| Address: South Cloisters, St |
| Luke's Campus |
| City, State, Province or County: |
| Exeter, Devon |
| Postal Code: EX1 2LU |
| Country: United Kingdom |
| Phone:+44 (0) 1392 72 2895 |
| Mobile: +44 (0) 788 144 8562 |
| Email: |
|
|
| Title: Researcher |
| Affiliation: Evidence-informed policy unit; University of Johannesburg |
| Address: Bunting Rd 5 |
| City, State, Province or County: Johannesburg |
| Postal code: 2006 |
| Country: South Africa |
| Phone: 0027 559 1909 |
| Email: |
ROLES AND RESPONSIBILITIES
Content: Katy-ann Daniel-Gittens and Alex Elwick Systematic review methods: Kristin Liabo and Jan Tripney Statistical analysis: Antonia Simon, Laurenz Langer Information retrieval: Kristin Liabo
SOURCES OF SUPPORT
The review is primarily conducted in our own time. Kristin Liabo is pursuing funding. CfBT Education Trust has agreed that their research officer A Elwick can have some allocated time to work on the full Campbell Review.
DECLARATIONS OF INTEREST
Two of the authors (A Simon and K Liabo) are co-authors of a published rapid review on this topic. With A Elwick, they have also published a perspectives paper based on that review.
PRELIMINARY TIMEFRAME
Approximate date for submission of the systematic review: June 2016
PLANS FOR UPDATING THE REVIEW
Kristin Liabo, as lead author, will be responsible for updates of the review.
AUTHOR DECLARATION
By completing this form, you accept responsibility for preparing, maintaining and updating the review in accordance with Campbell Collaboration policy. The Campbell Collaboration will provide as much support as possible to assist with the preparation of the review.
A draft review must be submitted to the relevant Coordinating Group within two years of protocol publication. If drafts are not submitted before the agreed deadlines, or if we are unable to contact you for an extended period, the relevant Coordinating Group has the right to de-register the title or transfer the title to alternative authors. The Coordinating Group also has the right to de-register or transfer the title if it does not meet the standards of the Coordinating Group and/or the Campbell Collaboration.
You accept responsibility for maintaining the review in light of new evidence, comments and criticisms, and other developments, and updating the review at least once every five years, or, if requested, transferring responsibility for maintaining the review to others as agreed with the Coordinating Group.
