Usability,learnability and performance evaluation of Intelligent Research and Intervention Software: A delivery platform for eHealth interventions

Abstract

Evaluation of an eHealth platform, Intelligent Research and Intervention Software was undertaken via cross-sectional survey of staff users and application performance monitoring. The platform is used to deliver psychosocial interventions across a range of clinical contexts, project scopes, and delivery modalities (e.g. hybrid telehealth, fully online self-managed, randomized control trials, and clinical service delivery). Intelligent Research and Intervention Software supports persuasive technology elements (e.g. tailoring, reminders, and personalization) as well as staff management tools. Results from the System Usability Scale involving 30 Staff and Administrative users across multiple projects were positive with overall mean score of 70 (“Acceptable”). The mean score for “Usability” sub-scale was 82 and for “Learnability” sub-scale 61. There were no significant differences in perceptions of usability across user groups or levels of experience. Application performance management analytics (e.g. Application Performance Index scores) across two test sites indicate the software platform is robust and reliable when compared to industry standards. Intelligent Research and Intervention Software is successfully operating as a flexible platform for creating, delivering, and evaluating eHealth interventions.

Keywords

eHealth intervention studies telehealth usability user experience

Introduction

Many eHealth interventions are developed in academic settings, unaffiliated with existing health care delivery systems. When research funding expires, the interventions are no longer supported.¹ Furthermore, costs to develop software and/or platforms to support eHealth intervention delivery are often prohibitive for researchers.^2,3 Thus, to date, many eHealth interventions are infrequently evaluated in trials⁴ or are evaluated without sufficient knowledge of their feasibility or scalability in the health care delivery system they were intended for.⁵

A small but promising literature has emerged for eHealth delivery platforms that can be configured to deliver concurrent interventions. Such platforms can introduce economies of scale by reducing the need for start-up costs for individual platforms per health care intervention. For example, an open-source platform LifeGuide⁶ and BeHealth Solutions/Manager⁷ have been used to deliver multiple self-managed eHealth interventions from varying clinical contexts (i.e. diabetes, insomnia, and alcohol exposure during pregnancy). To date, however, there are no published reports evaluating these platforms for software performance (i.e. reliability, responsiveness to usage demands) or usability evaluations of those administering the eHealth interventions via the platform. These are two critical dimensions of technology implementation that could negatively affect wider adoption and uptake. Scalability is highly dependent on reliable technology and user satisfaction, but staff usability evaluations are often not systematically reviewed or reported.^8,9

This article presents preliminary system performance results of an eHealth platform: Intelligent Research and Intervention Software (IRIS). This article addresses a gap in the literature by providing evidence of key eHealth platform performance indicators and usability assessments for different eHealth professional and administrative staff users.

IRIS platform overview

IRIS was developed as a generalizable platform initially deployed to support “Strongest Families Finland Canada” program (Canadian Institutes of Health Research; ClinicalTrials.gov no. NCT01750996) with the objective of providing participants with a secure, personalized website to access resources, complete interactive practice activities, and communicate directly with coaches. Additional partnerships were secured and multiple funded clinical trials assessing the effect of other IRIS powered interventions are currently underway. The not-for-profit Strongest Families Institute (SFI) currently uses the platform to help deliver health services at a distance to more than 2000 families.¹⁰ IRIS is currently used in 11 different applications (see Table 1) in three countries/languages¹¹ and is running on Version 4.4.10.

Table 1.

Focus of IRIS projects since beta development.

Launch	Name	Focus
2012	Finland	Population-based trial using the Strongest Families parenting program for caregivers of high-risk 4 year olds
2013	SFI	Psychologically informed education for caregivers of children with mild-to-moderate behaviors, anxiety and/or bedwetting problems. A large-scale service delivery system
2013	SHIFT	CBT for university students with symptoms of depression or anxiety
2013	DarWeb	Psychosocial intervention for pediatric recurrent abdominal pain
2013	SCAPI	Assessment tool measuring financial impact of health care
2014	Anxiety	Public health parent training for caregivers of children with anxiety symptoms
2014	PACT	Parent skills training for caregivers on communication with their adolescent
2014	Pain at home	Assessment tool to measure post-operative pain
2014	Breathe	Post-crisis CBT for youth 13–17 with anxiety
2014	FASD	Parent training for caregivers of children with FASD-related disruptive behavior
2014	CREO	Research skills training program

IRIS: Intelligent Research and Intervention Software; SFI: Strongest Families Institute; SHIFT: Self Help for Improving Feelings and Thoughts; CBT: cognitive behavioral therapy; SCAPI: Services for Children and Adolescents-Parent Interview: PACT: Parent Adolescent Communication Toolkit; FASD: fetal alcohol spectrum disorder; CREO: Comprehensive Research Education Online.

IRIS leverages open-source web technologies like Django and Backbone.js and is distributed as a Linux-based virtual appliance (VA), allowing it to be more easily deployed and maintained within different IT environments. The platform supports three major user types, each with a user interface (UI) or “view.” Participants (patients, clients) interact with intervention content through the Participant View, while staff (researchers, coaches, and clinicians) monitors participant progress through the Staff View. Technical staff accesses the site administrative controls and intervention authoring tools through the Administrative View. Intervention curriculum is organized into session folders and then each web page is sequenced and styled (see Figure 1).

Figure 1.

Examples of how two different IRIS Projects (Breathe and FASD) customized page layouts and styling through tools in the Administrative View.

IRIS supports the use of text, image, audio, video, image mapping, and a range of question/answer styles (e.g. multiple choice, true/false, and open-ended text boxes) to enable an interactive intervention curriculum. Additional features available include the following: automated, sequential participant ID number assignment; customizable “dashboards” that support information management for staff to view summary information quickly; automated task generation; and custom report generation with filtering capability.

Access to IRIS is password protected. Passwords are salted and hashed before being stored to ensure only authorized access. All web traffic is encrypted using Transport Layer Security/Secure Socket Layer (TLS/SSL). The IRIS VA is also protected with a software firewall with remote access possible only by authorized persons using the Secure Shell (SSH) protocol. The platform timestamps all account activity, which allows for comprehensive auditing. IRIS has been reviewed for compliance with personal health information (e.g. Personal Health Information Act (PHIA) in Nova Scotia, Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada) regulations and intervention studies conducted via IRIS have been formally approved by the institutional Research Ethics Board for each IRIS application/project.

Persuasive technology capabilities of the platform

As research has consistently demonstrated that persuasive technology (e.g. tailoring)^12–15 outperforms static health information delivery strategies for a variety of clinical contexts.^16–18 IRIS was designed with a range of interactive capabilities related to task support, dialogue and communication, and social support (see Table 2 for detailed list).

Table 2.

Persuasive technology capabilities supported by IRIS platform.

Persuasive design element	Examples from IRIS powered interventions
Primary task support
Reduction	Complex behaviors can be broken down into smaller tasks. Interactive fear ladder where youth initially identify main goals and then work through a series of prompts to help them create 5–8 smaller steps in that process
Tunneling	Delivers parenting-skills practice in sequential modules that can only be accessed once previous module is completed
Tailoring	Provides information adapted to youth based on whether they have comorbid substance abuse issues.
Personalization	Image mapping that allow users to choose if and what aspects of a topic they want to explore in more depth. Piping in the name of child and parent throughout the text
Self-monitoring	Provides diaries and tracking tools to view progress over time or to construct profiles of personal triggers and helpful strategies
Simulation	Provides built-in audio and video player so parents can listen to audio examples of other parents employing desirable communication strategies
Rehearsal	“Take-Away” homework activities stored in participant media library can be reviewed, edited or completed multiple times
Dialogue support
Reminders	Sends reminders to coaches and supervisors when follow-up is needed. Sends automated reminders and notifications to participants about upcoming assessments
Suggestions	Provides staff with decision support tree for assessing when to contact emergency services versus when to follow-up with health care provider
Similarity	Embedded supports directed to participants’ geographical location (e.g. local crisis help lines) or tailored content depending on user characteristics (e.g. children with divorced parents, youth who play sports)
Liking	Familiar navigation tools, ability to customize logos, color scheme, user friendly interface design for staff and participants
Social role	System takes on the role of a coach to automatically guide users step by step through activities
Social support
Social learning	Discussion boards, private messaging and “Ask the Expert” opportunities for participants to connect with para-professional coaches/research staff and other participants
Normative influence	Coaches flagged to review cases based on follow-up assessment results that have been compared to population norms
Recognition	Includes a “personal experience” page, opportunities for staff to safely share positive lessons learned
Cooperation	Ability for multiple staff members to review participant process within the intervention, discuss treatment plans and record progress notes to build common understandings
Social facilitation	Automated letters and emails to family health care providers or parents

IRIS: Intelligent Research and Intervention Software.

Each IRIS powered intervention/project can define how best to support their own population and staff needs. Task workflow design can be tailored to meet the objectives of the study/intervention program. For example, study group allocation processes (i.e. treatment/control groups) are designed in advance to securely regulate access to specific trial arm processes and task completion. Additionally, the system can be programmed to allow participants to modify information using an embedded inline editor (see Figure 2) or generate personalized and customized examples or skill demonstration media by “piping” information throughout program content (see Figure 3).

Figure 2.

Example of piping and transformational piping in IRIS. End user input and answers are piped into coaching scripts or later aspects of the workflow.

Figure 3.

Example of piping in personalized skill demonstration media throughout program content.

The platform captures a log of all activity (e.g. number of attempts at practice activities, time on task, and frequency of staff/participant contact) not just overall site analytics. This information can be fed back through the platform in real-time to adaptively support improved user engagement, both for participants and staff. Figure 4 provides an example of a real-time report that can quickly summarize participant progress and support staff task management.

Figure 4.

Example of a SFI care delivery staff report. A combination of participant and staff data was fed into a summary report template to help identify where participants and staff need additional support or where treatment fidelity or adherence issues are arising.

Reminders, notifications, tailored suggestions, and automated prompts can be configured to help users stay on track. Each project can assign participant and staff access rights to specific advanced communication features (e.g. file sharing, discussion boards, and “ask the expert”). Staff users can query information, generate custom reports (see Figure 3), and create personalized emails and letters. Selected datasets can be exported via standard comma-separated values (CSV) file format for additional analysis.

Objectives

Each IRIS powered project conducts usability testing relevant to their unique delivery context and end user participant population.¹⁹ To evaluate the platform globally as part of an internal quality control processes,²⁰ we pursued four key performance questions: (1) How “usable” and “learnable” is IRIS for Staff and Administrator users? (usability/satisfaction) (2) Do Staff and Administrative user groups differ in their perceptions of IRIS’ usability and learnability? (3) How much site downtime do IRIS users encounter and how quickly does the software recover (availability)? and (4) How often do IRIS users experience lagging and slow response times when visiting IRIS powered sites (responsiveness)?

Methods

A combination of system performance monitoring and cross-sectional survey design²¹ were employed to develop a more comprehensive view of platform stakeholders needs and focus on aspects of the platform that could be enhanced during future development.²²

Measures

The System Usability Scale

The System Usability Scale (SUS) is a highly reliable tool^23,24 designed to solicit subjective feedback on overall usability and user satisfaction. The questionnaire provides 5-point Likert scale response options from 1 (Strongly disagree) to 5 (Strongly agree) for 10 items. Items 1, 3, 5, 7, and 9 are positively worded and items 2, 4, 6, 8, and 10 are negatively worded. The SUS has been demonstrated to be effective in discriminating good and bad usability features even with small sample sizes (<10). Rating categories for SUS scores are as follows: 0–64 is unacceptable, 65–84 is acceptable, 85–100 is excellent, with a score of 82 representing the likelihood to recommend (LTR) threshold Promoters (i.e. people likely to recommend your product) have an average SUS score of 82, whereas Detractors (i.e. people who would not recommend a product) have an average SUS score of 67.²⁵ Factor analysis of SUS item structure supports extracting additional information through two sub-scales: an 8-item “Usability” and 2-item “Learnability” scale.²⁶

Application performance management (APM).

New Relic is a software-as-a-service (SaaS) third-party plugin that can be integrated into a web application to monitor key application performance metrics.^27–29 New Relic (Lite Tier) tracked the following performance indicators:

Availability (i.e. % of time the IRIS applications were able to be accessed; also known as “uptime”),

Load time (i.e. the average amount of time (in seconds) it takes a page to load on the site, from initiation of the page view (e.g. click on a page link) to load completion in the browser), and

Application Performance Index (Apdex).³⁰ Apdex is an open standard³¹ for reporting and comparing the performance of software applications. The Apdex converts raw use data into a uniform metric to help analyze the degree to which the software’s performance meets user expectations. The Apdex target time [T] was set at the industry standard 5 s where all response times that took less than the target 5 s were considered “satisfactory” to the user (i.e. (Satisfied Count + Tolerating Count/2)/Total Samples). Based on research, users will tolerate response times that are up to 4 times the target time. Thus, response times greater than 5 s but less than 20 s were considered “tolerated” by users, and those greater than 20 s were considered “frustrating” to users. Apdex scores are assigned the following ratings: a score of 0.94–1.00 = excellent, 0.85–0.93 = good, 0.70–0.84 = fair, 0.50–0.69 = poor, and 0.00–0.49 = unacceptable.³²

Procedures

SUS

Criterion sampling,³³ a purposive sampling strategy frequency used in quality assurance evaluations was used to identify a pool of potential respondents. The criterion for inclusion was that respondents had to have used IRIS as a Staff or Administrative user within the previous 6 months (May 2014–October 2014). A total of 44 users were identified and asked to complete the SUS online via SelectSurvey as well as project-based questions. The two user groups were included because they perform different roles within each IRIS intervention and interact with the platform differently. Administrators author and configure intervention workflow prior to deployment while Staff (researchers, coaches, and clinicians) monitor treatment progress and manage staff tasks during intervention delivery. Using a modified Dillman³⁴ protocol, all potential respondents were emailed a link to the questionnaire and asked to complete the survey within 1 week. An email reminder was sent 2 days before the deadline.

Application performance management (APM)

Two of the 11 IRIS sites were purposively selected for application performance monitoring via New Relic. These two test sites (see Table 1), Breathe and fetal alcohol spectrum disorder (FASD)), were selected because (1) they were the most recently deployed sites on IRIS Version 4.0.1, (2) they incorporated more persuasive technology features than other sites (i.e. more complex tasks would be performed), and (3) were expected to have the highest number of active users (i.e. higher site traffic) during the testing period. Results from the two selected sites would provide the most conservative estimate of how IRIS was performing under the heaviest expected workload. There are no established recommendations for establishing performance baseline of network and software although the common practice is to review metrics at least over a 7-day period.³⁵ The authors agreed that 3 weeks per month of continuous monitoring provided a more robust analysis of performance over time—particularly since test case sites were only recently deployed and levels of use were expected to vary. New Relic (Lite Tier) was installed on both sites providing continuous monitoring of the application between 8th September and 5th October 2014. Detailed weekly analytics summaries were provided for free by New Relic.

Analysis

Descriptive analysis of demographic and individual SUS questionnaire items were conducted in SPSS (Version 21) to examine measures of central tendency and variability across user groups (i.e. Staff and Administration) and level of use (e.g. frequent users and infrequent users). Individual participant’s responses were converted to 0–100 percentiles as per SUS guidelines³⁶ and scored by comparing them to standard rating scales.³⁷ The mean SUS score for all participants was calculated. LTR was determined by comparing overall mean SUS score to industry thresholds (e.g. a SUS of 80 represents the “likelihood to recommend” threshold). Weekly performance analytics provided by New Relic were compiled in SPSS and examined for variability.

Results

Demographics

A total of 33 Staff and Administration users completed the SUS questionnaire (response rate: 61%). Three respondents who each had 1 item with missing data on the SUS portion of the survey were excluded from analysis. In all, 23 percent of included respondents (n = 7) were identified as “Administrators” and 77 percent (n = 23) as “Staff” (see Table 3). There was large variability in the length of time respondents had been using IRIS. The mean number of months using the platform was 15.67 (standard deviation (SD) = 10.94; 95% confidence interval (CI): 11.34–20.00, n = 27). Two respondents had only been using the system for 3 months while four others had been using the system for 30 months or more. The sample was also diverse in terms of the frequency with which users interacted with IRIS. In all, 73 percent (n = 22) of the participants reported using IRIS “daily” and 13 percent (n = 4) used it less than once a week.

Table 3.

Demographic variables of SUS respondents.

Characteristic	Staff (n = 23)	Administrators (n = 7)
	N (%)	N (%)
Length of time using IRIS^a,b
Less than 12 months	9 (43)	1 (17)
12–23 months	10 (48)	2 (33)
24 months or more	2 (9)	3 (50)
Frequency of use
Daily	18 (78)	4 (57)
2–3 times per week	1 (4)	1 (14)
Once a week	1 (4)	1 (14)
1–2 times per month	2 (9)	1 (14)
Less than once per month	1 (4)	0 (0)

SUS: System Usability Scale; IRIS: Intelligent Research and Intervention Software.

One administrator had missing data for this item so percentages were calculated from total n = 6.

Two staff had missing data for this item so percentages were calculated from total n = 21.

System usability and satisfaction

Overall, Staff and Administration users felt IRIS performed well. The group mean for overall SUS score was 70 an “Acceptable” rating based on standard SUS.³⁸ The industry benchmark SUS score of 80 for users to more likely be “promoters” of your product was not met. The mean score for the “Usability” sub-scale was 82 and the mean score for the “Learnability” sub-scale was 61. Individual item means are reported in Table 4. The majority of users had a favorable opinion about IRIS in terms of how confident they felt using the system (Item 9) and ease of use (Items 3 and 7). The need for technical support (Item 4) and the need to learn a lot of things before using the system (Item 10) were two items in which respondents indicated concern. Independent samples t-tests were conducted to compare individual SUS scores for Staff and Administrator groups and for users with less than 12 months experiencing using IRIS and those with more 12 months or more. There was no significant difference in SUS scores for Staff (M = 70.2, SD = 15.2) and Administrator (M = 67.5, SD = 20.5) user groups; p = 0.705. There was no significant difference in SUS scores for those using IRIS for less than 12 months (M = 72.5, SD = 16.8) and those using it for 12 months or more (M = 68.1, SD = 16.1); p = 0.496.

Table 4.

Summary of SUS questionnaire results for overall sample of Staff and Administrator user groups.

SUS items^a		Mean	95% CI	SD	Median	Minimum	Maximum
1	I think that I would like to use this system frequently	3.87	3.54–4.19	0.86	3.90	2	5
2	I found the system unnecessarily complex	2.27	1.88–2.66	1.05	2.11	1	5
3	I thought the system was easy to use	4.03	3.73–4.34	0.81	4.12	2	5
4	I think that I would need the support of a technical person to be able to use this system	2.33	1.94–2.73	1.06	2.24	1	4
5	I found the various functions in this system were well integrated	3.70	3.40–4.00	0.79	3.75	2	5
6	I thought there was too much inconsistency in this system	2.40	2.01–2.79	1.04	2.39	1	4
7	I would imagine that most people would learn to use this system very quickly	3.90	3.62–4.18	0.76	3.96	2	5
8	I found the system very cumbersome to use	2.10	1.74–2.46	0.96	1.96	1	5
9	I felt very confident using the system	3.97	3.70–4.23	0.72	4.00	2	5
10	I needed to learn a lot of things before I could get going with this system	2.53	2.14–2.92	1.04	2.45	1	5

SUS: System Usability Scale; CI: confidence interval; SD: standard deviation.

Items 2, 4, 6, 8, and 10 are negatively worded. Lower Means for these items represent higher perceived satisfaction.

Performance monitoring

IRIS availability experienced by users was high with the sites being available 98.85 percent (Site 1) and 97.06 percent (Site 2) of the time during the testing phase. Overall performance measures are summarized in Table 5. Weekly averages for Load times never exceeded 1.61 s. Weekly Apdex scores (efficiency) ranged from 0.99 to 1.00 (“Excellent” rating) indicating the application responded to users within 5 s 99–100 percent of the time. These threshold response times did not vary significantly across sites, with the widest weekly discrepancy between Sites 1 and 2 being only 0.86 s. Apdex ratings were consistent across 3 weeks of the testing period confirming reliability over time.

Table 5.

Results of performance monitoring metrics for two IRIS sites over 3 week testing.

	Test Site 1			Test Site 2
	Week 1	Week 2	Week 3	Week 1	Week 2	Week 3
Uptime/availability	100%	99.94%	98.33%	97.93%	99.63%	99.60%
Average load time	0.91 s	0.87 s	0. 75 s	1.32 s	1.36 s	1.61 s
Apdex score	1.00*[T]	0.99*[T]	1.00*[T]	0.99*[T]	0.99*[T]	0.99*[T]

IRIS: Intelligent Research and Intervention Software; Apdex: Application Performance Index.

A small report group is defined as one having fewer than 100 samples and is indicated by including an (*) next to the [T] score.

Discussion

Performance problems and perceived eHealth platform barriers need to be addressed before they threaten an interventions viability and scalability.³⁸ The average overall SUS score across Staff and Administration participants in this study was positive (an average SUS score of 70), but slightly below the “likely to recommend” threshold. However, sub-scale analysis revealed a significant difference in overall perceptions of “Usability” (mean score of 82) compared to overall perceptions of “Learnability” (mean score of 61). This suggests that while usability perceptions were quite good, more diagnostic analysis of Staff and Administrative user training and technical support needs is warranted.

Although research has consistently shown that SUS scores are correlated to length of time using a product, our study did not find any significant differences related to length of time using IRIS. The SUS provides a reliable rating of general usability and user satisfaction but is not a diagnostic tool for identifying specific product improvements to be undertaken. As Administrators and Staff use IRIS differently in each project, we will need to balance these global findings with project-specific variables that may affect usability assessments. While the generalizability of these findings may be limited to the specialized audience of IRIS projects, they do point to needed future work in exploring learnability and training factors that are a function of both the platform and implementation context.

Examining standard web-performance metrics for eHealth interventions needs to be a routine part of usability testing. The public’s high expectations for fast and engaging web experiences are becoming critical factors for positive health intervention outcomes^39,40 but are underreported and overlooked measures in the majority of eHealth intervention research. Our preliminary examination of technical performance across two test sites indicated that IRIS is a responsive, efficient and reliable platform for end users. The advanced data analytics tools provide important data about what users experience in eHealth interventions from a technical standpoint and how that can be mitigated so as not to interfere with treatment adherence, outcomes and long-term sustainability.⁴¹

Although the SUS is reliable even with small sample sizes, not all IRIS Staff and Administrators who have worked on the platform responded to requests for feedback, introducing greater potential for rater bias.⁴² By including multiple user groups with a range of experience with IRIS, we believe the sample is still representative. Finally, performance monitoring involved a cross-sectional snapshot of only two test sites over a 3-week period. Although this time period met recommendations we feel concurrent, longitudinal analysis over multiple IRIS sites will help us better understand how the system performs under different conditions and usage loads. We plan to use these results to develop an ongoing performance monitoring plan for testing IRIS under different conditions. The free version of New Relic used in this analysis did not allow us to parse out the causes of downtimes (i.e. network failure vs server failure vs software failure)⁴³ but those analytics are available for subscribed users. Performance monitoring will be incorporated into IRIS site usability analysis moving forward.

Conclusion

Preliminary evaluations of IRIS through performance monitoring and usability analysis suggest it is a reliable and learnable environment for staff and administrator users where eHealth interventions can be deployed with minimal software performance errors. Many IRIS powered interventions have only recently been deployed promising greater capacity in the future to examine usability across interventions and clinical populations. Future interdisciplinary work should focus on the intersection of traditional usability and human factors research with best-practices for software performance monitoring. The “dependability” and “usability” of systems are not isolated constructs. A clearer picture of how these core attributes are dynamically related remains a vital area of future research.

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Drs. McGrath and Lingley-Pottie may incorporate the IRIS platform in the future.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project is funded by an operating grant from the Canadian Institutes of Health Research (CIHR No.103146) and an Innovacorp Early Stage Commercialization Fund grant. Dr Amanda Newton holds a CIHR New Investigator Award. Dr Patrick McGrath holds a Canada Research Chair (Tier 1) in pediatric pain.

References

Glasgow

Phillips

Sanchez

MA.

Implementation science approaches for integrating eHealth research into practice and policy. Int J Med Inform. Epub ahead of print 30 July 2013. DOI: 10.1016/j.ijmedinf.2013.07.002.

Van Velsen

Wentzel

Van Gemert-Pijnen

JEWC

. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc 2013; 2(1): e21.

Tate

Finkelstein

Khavjou

. Cost effectiveness of Internet interventions: review and recommendations. Ann Behav Med 2009; 38(1): 40–45.

Christensen

Petrie

State of the e-mental health field in Australia: where are we now?

Aust N Z J Psychiatry 2013; 47(2): 117–120.

Makai

Melis

Olde-Rikkert

MG.

Technical difficulties and evaluating e-health interventions. JAMA Intern Med 2014; 174(2): 304–305.

Williams

Yardley

Wills

GB.

A qualitative case study of LifeGuide: users’ experiences of software for developing Internet-based behaviour change interventions. Health Informatics J. 2013; 19(1): 61–75.

Thorndike

Ritterband

Gonder-Frederick

. A randomized controlled trial of an Internet intervention for adults with insomnia: effects on comorbid psychological and fatigue symptoms. J Clin Psychol 2013; 69(10): 1078–1093.

Van Gemert-Pijnen

Nijland

Van Limburg

. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011; 13(4): e111.

Brewster

Mountain

Wessels

. Factors affecting front line staff acceptance of telehealth technologies: a mixed-method systematic review. J Adv Nurs 2014; 70(1): 21–33.

10.

E-Mental Health in Canada: transforming the mental health system using technology. Ottawa, ON, Canada: Mental Health Commission of Canada, 2014, http://www.mentalhealthcommission.ca

11.

McGrath

Sourander

Lingley-Pottie

. Remote population-based intervention for disruptive behavior at age four: study protocol for a randomized trial of Internet-assisted parent training (Strongest Families Finland-Canada). BMC Public Health 2013; 13: 985.

12.

Kelders

Kok

Ossebaard

. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012; 14(6): e152.

13.

Lehto

Oinas-Kukkonen

Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature. J Med Internet Res 2011; 13(3): e46.

14.

Fogg

BJ.

Persuasive technology: using computers to change what we think and do. Boston, MA: Morgan Kaufmann Publishers, 2003.

15.

Oinas-Kukkonen

Harjumaa

Persuasive systems design: key issues, process model, and system features. Commun Assoc Inf Syst 2009; 24(1): 485–500.

16.

Lustria

MLA

Noar

Cortese

. A meta-analysis of web-delivered tailored health behavior change interventions. J Health Commun 2013; 18(9): 1039–1069.

17.

Lustria

MLA

Cortese

Noar

. Computer-tailored health interventions delivered over the web: review and analysis of key components. Patient Educ Couns 2009; 74(2): 156–173.

18.

Noar

Benac

Harris

MS.

Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007; 133(4): 673–693.

19.

Wozney

Newton

Perri

. Intelligent Research and Intervention Software (IRIS): an eHealth platform for behavioral interventions. In: Proceedings of the 16th international symposium on health information management research: exploiting health informatics for connected, collaborative and customized patient care (ed Abidi

SSR

Bath

), Halifax, NS, Canada, 27–28 June 2013, pp. 159–160. Dalhousie University and University of Sheffield.

20.

Damschroder

Aron

Keith

. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4(1): 50.

21.

Bordens

Abbott

BB.

Research design and methods: a process approach. New York: McGraw-Hill, 2011.

22.

Walji

Kalenderian

Piotrowski

. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014; 83(5): 361–367.

23.

Bangor

Kortum

Miller

JT.

An empirical evaluation of the System Usability Scale. Int J Hum Comput Interact 2008; 24(6): 574–594.

24.

Lewis

Sauro

. The factor structure of the system usability scale. In: Kurosu

(ed.) Human centered design. Berlin: Springer, 2009, pp. 94–103.

25.

Sauro

Measuring usability: 10 things to know about the System Usability Scale (SUS), http://www.measuringusability.com/blog/10-things-SUS.php (18 June 2013, accessed 12 October 2014).

26.

Lewis

Sauro

. The factor structure of the System Usability Scale. In: Proceedings of the human computer interaction international conference (HCII 2009), San Diego, CA, 19–24 July 2009.

27.

Farnaz

Izadyan

Fricker

. Analytics for product planning: in-depth interview study with SaaS product managers. In: 2013 IEEE 6th international conference on cloud computing (CLOUD), Santa Clara, CA, 28 June–3 July 2013, pp. 871–879.

28.

Yahoo Finance. New relic to bring SaaS-based app performance monitoring to windows Azure developer portal, http://finance.yahoo.com/news/relic-bring-saas-based-app-160000220.html (2012, accessed 15 September 2014).

29.

Chiou

Chin-Chao

Chyuan

A strategic framework for website evaluation based on a review of the literature from 1995–2006. Inform Manag 2010; 47(5): 282–290.

30.

Sevcik

Defining the application performance index. Business Communications Review, March, http://apdex.org/docs/Defining_The_Application_Performance_Index.pdf (2005, accessed 12 October 2014).

31.

International Telecommunications Union. Definition of “Open Standards”, http://www.itu.int/en/ITU-T/ipr/Pages/open.aspx (2005, accessed October 12 2014).

32.

Gunther

. The Apdex Index revealed. CMG MeasureIT, February 2009, http://www.cmg.org/publications/measureit/2009-2/mit56/measureit-issue-7-02-the-apdex-index-revealed/

33.

Patton

Qualitative evaluation and research methods. Beverly Hills, CA: SAGE, 1990, pp. 169–186.

34.

Dillman

DA.

Mail and telephone surveys: the total design method. New York: Wiley Interscience, 1978.

35.

Haletky

EA.

Using application performance management for security. New Relic, https://try.newrelic.com/rs/newrelic/images/NewRelic-Using_APM_for_Security.pdf (2012, accessed 12 October 2014).

36.

Bangor

Kortum

Miller

Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

37.

McLellan

Muddimer

Peres

CS.

The effect of experience on System Usability Scale ratings. J Usability Stud 2012; 7(2): 56–67.

38.

Sauro

A practical guide to the System Usability Scale: background, benchmarks, & best practices. Denver, CO: Measuring Usability LLC, 2011.

39.

Schneider

Van Osch

De Vries

Identifying factors for optimal development of health-related websites: a Delphi study among experts and potential future users. J Med Internet Res 2012; 14(1): e18.

40.

Moreno

del Castillo

JMM

Porcel

. A quality evaluation methodology for health-related websites based on a 2-tuple fuzzy linguistic approach. Soft Comput 2010; 14(8): 887–897.

41.

Sang-Myung

Ungson

Russo

MV.

What determines an engaging website? An empirical study of website characteristics and operational performance. J High Tech Manag Res 2011; 22(1): 67–79.

42.

Wood

TJ.

Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract 2014; 19: 409–427.

43.

Sandeep

Chhillar

RS.

Empirical study of root cause analysis of software failure. ACM SIGSOFT 2013; 38(4): 1–7.