Using Google Analytics to Evaluate the Impact of the CyberTraining Project

Abstract

A focus on results and impact should be at the heart of every project's approach to research and dissemination. This article discusses the potential of Google Analytics (GA: http://google.com/analytics) as an effective resource for measuring the impact of academic research output and understanding the geodemographics of users of specific Web 2.0 content (e.g., intervention and prevention materials, health promotion and advice). This article presents the results of GA analyses as a resource used in measuring the impact of the EU-funded CyberTraining project, which provided a well-grounded, research-based training manual on cyberbullying for trainers through the medium of a Web-based eBook (www.cybertraining-project.org). The training manual includes review information on cyberbullying, its nature and extent across Europe, analyses of current projects, and provides resources for trainers working with the target groups of pupils, parents, teachers, and other professionals. Results illustrate the promise of GA as an effective tool for measuring the impact of academic research and project output with real potential for tracking and understanding intra- and intercountry regional variations in the uptake of prevention and intervention materials, thus enabling precision focusing of attention to those regions.

Introduction

Bully/victim problems among school pupils is an international problem.¹ A robust cross-national knowledge base of intervention and prevention programs in the area has been developed and illustrated.^2,3 Coupled with this increased knowledge regarding traditional bully/victim problems (also referred to as “face-to-face” or “f2f” bullying⁴), we are also at the stage where we are increasing our knowledge of new, more contemporary forms of such problems, such as cyberbullying.^4–10 However, unlike f2f bully/victim problems, we have scant knowledge of how to counter the insidious effects of cyberbullying.

The recently completed CyberTraining project (funded by the EU Lifelong Learning Programme: [Project No.142237-LLP-1-2008-1-DE-LEONARDO-LMP] http://cybertraining-project.org) has provided a well-grounded, research-based training manual on cyberbullying for trainers. The manual was based upon the outcomes of: (a) a needs assessment conducted with trainers across partner countries (Germany, Ireland, Portugal, Spain, United Kingdom, and Bulgaria), (b) the perspectives solicited from subject experts via the Delphi method, and (c) transnational analyses of the situation in partner countries, in terms of research efforts and interventions.

The manual includes background information on cyberbullying, its nature and extent in Europe, current projects, initiatives and approaches tackling the cyberbullying problem, best practice Europe-wide, as well as practical guidance and resources for trainers working with the target groups of: (a) pupils, (b) parents, and (c) teachers, schools, and other professionals. The manual concludes with a comprehensive compilation of references, links, and other resources for trainers. The manual was published online in Autumn 2010 in the user-friendly format of an eBook and is available in English, Spanish, French, German, Portuguese, and Bulgarian versions (see also Mora-Merchan and Jäger¹¹ for a presentation of key reports from the project partner countries as well as country reports from Central and Eastern Europe, the Mediterranean countries, Scandinavia, Australasia, and the United States).

Effective project dissemination and measuring project impact

A project is only as useful as the scope and impact of its dissemination. Researchers and practitioners focus predominantly on traditional routes of dissemination, such as conference presentations, peer-reviewed journal articles, monographs, book chapters, and static Web content. However, in a climate of evidence-informed practice, commissioners of research and intervention/prevention programs are seeking a more robust analysis of dissemination activity—one that does not simply follow the traditional top-down approach. Rather, what is increasingly required is an approach that allows analysis of impact and usability by end-users. Considering that much research output and the content of intervention/prevention programs are made available via the Internet and Web 2.0 technologies, it is incumbent upon researchers and practitioners to understand how and where the information is being accessed. For example, while most governments have online information regarding bully/victim prevention and helpful resources, the online monitoring of Web-traffic pertaining to those materials would give an early indication of hot regional activity, thus enabling the easy identification of geographical areas that could benefit from rapid communication and dissemination of information (e.g., alert schools in that area to be increasingly vigilant and resources available).

One such tool, Google Analytics (GA) is well positioned to serve those who seek such intelligence. GA (www.google.com/analytics), one of the most sophisticated Web analytics tools,¹² is a free service that provides detailed information and statistics about Website visitors and their activity, such as how far into the Website they travel and what links they click on most frequently (useful for search engine optimization). For example, design changes based on information compiled from GA greatly improved the Rutgers-Newark Law Library for the Center of Law and Justice's Website, bringing it more Web traffic, greater user loyalty, and more effective navigation.¹³

Aim of the current study

The central aim of the present study was to explore the potential of GA as an effective resource for measuring the impact of academic and practitioner research output. Specifically, the study explored the functionality of GA as a complimentary resource in measuring the impact of the CyberTraining Project by identifying when users were interacting with online resources, while also identifying the origins of all access.

Method

One more of most sophisticated aspects of GA V2 is the new immersive data interaction model. The V2 User Interface is completely customizable, ranging from digital dashboards (see Fig. 1 below) to overview reports and presentation of the data. Presented in Figure 1 below and in the graphs and statistics that follow are examples of the reports that we used, including site usage/visits, page views, bounce rates, average time on site, and the percentage of new visits.

FIG. 1.

Sample dashboard with three sections. First section contains a line graph representing Site Usage statistics. Third section representing the Map Overlay and third section representing a Visitors Overview.

Chronologically, Figure 1 presents the usability data on a monthly basis (also available on a daily and weekly basis). This unique facility allows for comparison of data from different date ranges—for example, before, during, and after dissemination of the project.

Presented in Figure 2 below is further data intelligence, specifically focused on an overview of site visits. This data provides specific information on usability, such as user loyalty, which includes variables such as the number of visits, unique visitors to the site, and the average time spent on the site by each visit.

FIG. 2.

Visitors Overview. A line graph representing salient information with regard to visitors to the site, including Page Visits, New Visits (percent), Average Time on Site, Absolute Unique Visits, Average Page Views, Time on Site, Bounce Rate.

Figure 3 below provides added insight into the source of all traffic, in terms of direct traffic, referring sites, and search engines.

FIG. 3.

Traffic Sources Overview. A line graph representing the source of visits, including Direct Traffic (percent), Referring Sites, and Search Engine Access.

Presented in Figure 4 below is further data intelligence that illustrates clearly the sources of all access that is related to the overall visits to the site.

FIG. 4.

Statistics representing the geographical origins of access to the site with associated variables, such as Page Visits, New Visits (percent), Average Time On Site, Absolute Unique Visits, Average Page Views, Time On Site, Bounce Rate.

Results

Table 1 below represents sequential data points across four distinct time frames: (a) 1 month predissemination, (b) dissemination, (c) 1 month postdissemination, and (d) 3–5-month follow-up. Variables monitored included: Visits, Pages Visited, New Visits (percent), Average Time on Site, Absolute Unique Visits, Direct Traffic (percent), Referred Traffic (percent), Search Engines, Primary Sources, Secondary Sources, Top Four Countries, and Top Content.

Table 1.

Chronological Data Captures

	Predissemination	Dissemination	Postdissemination	Followup
Variable measured	Aug.–Sept. 2010	Oct.–Nov. 2010	Nov.–Dec. 2010	Mar.–May 2011
Visits	227	568	630	1,975
Pages visited	5.47	2.68	2.88	2.48
New visits (%)	59.47	66.90	71.75	61.37
Avg. time on site	03:40	02:17	03:12	02:53
Absolute unique visits	140	415	495	1270
Direct traffic (%)	59.03	37.62	41.75	43.29
Referring sites (%)	18.06	24.12	22.86	33.92
Search engines (%)	22.91	38.26	35.40	22.78
Primary source	Direct	Search engines	Direct	Direct
	59.03%	38.26%	41.75%	43.2%
Secondary source	Google	Direct	Google	Referring Sites
	22.47%	37.62%	35.40%	34.07%
Top 4 countries	1. Bulgaria	1. Germany	1. Germany	1. Germany
	2. Germany	2. Italy	2. United States	2. Portugal
	3. United Kingdom	3. United States	3. United Kingdom	3. United States
	4. Switzerland	4. Slovakia	4. Bulgaria	4. Italy

Interpretation

During the predissemination phase, 59.67 percent of the 227 visitors to the site during the specified time period were new users. The average time on site was 00:03:40. Usefully, direct traffic (someone entering the specific URL as opposed to using a search engine) to the site was 37.62 percent compared to 18.06 percent for referred access. Over one-fifth (22.47 percent) of the referred traffic came through Google's search engine. An average of 5.67 pages was visited during each visit. In a descending order, the top access countries to the site were (a) Bulgaria (the Web developers), (b) Germany (project management), (c) the United Kingdom, and (d) Switzerland. It would be expected that these figures should increase as information about the CyberTraining Website is disseminated across multinational channels.

During the dissemination phase, 568 people visited the site, an increase of 341. Two-thirds (66.90 percent) of users were new to the site. On average, 2.68 pages were accessed during each visit, with the average time on site reduced to 00:02:17. Direct traffic to the site was 59.03 percent, with 24.12 percent being referred access. In a descending order, the top access countries to the site were (a) Germany, (b) Italy, (c) United States, and (d) Slovakia. As mentioned above, one would expect in the future that users would spend more time on the site as a result of additional content.

During the postdissemination phase, 630 people had visited the site during the specified time period— an increase of 62. Nearly three-quarters of users (71.75 percent) were new to the site. On average, 2.88 pages were accessed during each visit, with the average time on site being 00:03:12. Direct traffic to the site was 41.75 percent with 22.86 percent being referred access. In a descending order, the top access countries to the site were (a) Germany, (b) United States, (c) the United Kingdom, and (d) Bulgaria. Again with the addition of new material and the uploading of the complete training manual, one would expect more new users and increased time on site/increased number of pages accessed.

At follow-up, 1,975 people had visited the site during the specified time period—61.37 percent of users being new to the site. On average, 2.48 pages were accessed during each visit. The average time on site was 00:02:53. Direct traffic to the site was 43.29 percent, with 33.92 percent being referred access.

Discussion

The aim of the research was to ascertain the efficacy of GA as an effective resource for measuring the impact of the CyberTraining project. As part of the project's dissemination strategy, the eBook was uploaded to the project Website in each of the partner country languages. A focused campaign of project dissemination via traditional academic means (e.g., conference presentations, book) promoted the project outside of the partner countries. As noted, such traditional approaches to project dissemination are one-way. With an increasing focus on evidence-informed results, the focus here on using GA to further understand the uptake of the project training manual was positive—especially considering that many of the project team are involved in another EU-funded project that builds upon the results of this project (CyberTraining-4-Parents [CT4P]: Project number: 510162-LLP-1-2010-1-DE-GRUNDTVIG-GMP). Similar large-scale multipartner projects in the area (e.g., COST Action IS0801: http://sites.google.com/site/costis0801) could also benefit from the use of GA in their attempts to influence research, policy, practice, and outcomes.

In essence, GA has demonstrated its worth in assessing the early impact of the CyberTraining project and has allowed for the project team to further understand how best to optimize the product (i.e., the Website and the eBook) for ease of access and navigation by unique and referred users. Direct access was, on average 45 percent, representing informed users. Considering that the content of the eBook was determined by a robust research process, this finding was heartening. Indeed, this figure may also account for informed users who learnt about the project through the traditional means of dissemination also employed. On average, 30 percent of new users accessed the Website and eBook via Google's search engine—a finding useful for further Web optimization. Such work should evidence an increase in site traffic from uninformed users and the general public—a key market for the content of the eBook. In a similar vein, with referred access rates of between 20 percent and 25 percent from other Websites, link-building is an issue that the project team should now also consider. There may also be some potential in developing social networking marketing campaigns (e.g., Facebook).

Such information proves valuable in the iterative development and dissemination of the project and has, directly, informed the planning of the new CT4P project. In terms of cybersafety, advice for parents¹⁴ is that: “if you don't understand it, you can't teach it!” The results here from GA further this: if you can't find it, you can't use it!

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Smith

, Morita

, Junger-Tas

et al. 1999. The nature of school bullying: a cross national perspective. London and New York: Routledge.

Farrington

, Ttofi

MM.

School-based programs to reduce bullying and victimization. Campbell Systematic Reviews, 2009; 6. 10.4073/csr.2009.6.

Smith

, Pepler

, Rigby

. 2004. Bullying in schools: how successful can interventions be? Cambridge: Cambridge University Press.

Mc Guckin

, Cummins

, Lewis

. f2f and cyberbullying among children in Northern Ireland: data from the Kids Life and Times surveys. Psychology, Society, & Education, 2010; 2:67–78.

Campbell

. Cyber bullying: an old problem in a new guise? Australian Journal of Guidance and Counselling, 2005; 15:68–76.

Patchin

, Hinduja

. Bullies move beyond the schoolyard: a preliminary look at cyberbullying. Youth Violence and Juvenile Justice, 2006; 4:148–169.

Smith

, Mahdavi

, Carvalho

et al. 2006. An investigation into cyberbullying, its forms, awareness and impact, and the relationship between age and gender in cyberbullyingResearch Brief No. RBX03-06London: DfES.

Ybarra

, Mitchell

. Prevalence and frequency of Internet harassment instigation: implications for adolescent health. Journal of Adolescent Health, 2007; 41:189–195.

Ybarra

, Mitchell

. Online aggressor/targets, aggressors, and targets: a comparison of associated youth characteristics. Journal of Child Psychology and Psychiatry, 2004; 45:1308–1316.

10.

Ybarra

, Mitchell

, Wolak

et al. Examining characteristics and associated distress related to Internet harassment: findings from the second youth Internet safety survey. Pediatrics, 2006; 118:1169–1177.

11.

Mora-Merchan

, Jäger

. 2011. Cyberbullying: a cross-national comparison. Landau: Verlag Emprische Pädagogik.

12.

Dyrli

. How effective is your Website? Free online tools help measure site success. District Administration, 2006; 42:72.

13.

Fang

. Using Google Analytics for improving library Website content and design: a case study. Library Philosophy and Practice, 2007 June(special issue on Libraries and Google).

14.

Mc Guckin

, Cummins

, Lewis

. New bottle: old wine! Persistence of school bullying among primary school pupils and the emergence of cyberbullying. ARK Research Update, 2010; 71Belfast: ARKwww.ark.ac.uk/publications/updates/update71.pdf