A Critical Review of User Studies on Healthy Smart Homes

System Effectiveness Evaluation

One of the popular themes in healthy smart home projects is to develop predictive systems of human behaviours from monitoring data, enabling the environment to be aware of the activities [25]. Accordingly, various monitoring systems have been proposed, and sensors become local intelligent units for the detection of changes in human behaviour and health conditions. Helal et al. [24] conducted an experiment to check the accuracy of the in-door tracking system using a test dummy, Matilda, in a laboratory mock up house. Matilda’s house is a simulation instrument with ultrasonic location tracking sensors that can run for weeks according to a generated trace of typical elderly daily activities. They classified errors into four categories (absolute, environmental, reflective and resonant) and then conducted 1356 measurements in 22 selected positions. The accuracy of the sensed locations was judged by comparing them with the real measured positions. Their achieved error rate of 22 cm was not tolerable and thus needed to be reduced using error correction algorithms. Based on the belief that computer vision can collect more in-depth information, Mihailidis et al. [25] emphasized the use of computer vision in a sensing agent for an intelligent environment that assists older adults with dementia during the activities of daily living (ADL). They conducted several experiments to verify the accuracy and repeatability of the agent. For the task, hand washing was chosen because the model of the required agents was relatively simple, older adults with dementia often would not complete the task properly, and the task was relatively safe for clinical trials. They argued that the level of accuracy and repeatability observed for the sensing agent would be sufficient for the overall system, allowing appropriate decisions to be made.

Much research on user-activity assistance applications has used the location of users as context. However, Isoda et al. [26] employed a time sequence aspect to describe the user’s contexts. The proposed system discriminates user states by employing user representations to express both spatial and temporal relationships between humans and objects in an environment. To validate the performance of the prototype system, they conducted tests in an experimental house containing various sensors and the Radio-Frequency Identification (RFID) tags, concerning four types of user-state classes – not in the kitchen; in the kitchen, but only momentarily; in the kitchen continuously for a brief period (less than 10 seconds); in the kitchen continuously for a long period (at least 10 seconds). For effective monitoring of changes in health status, Tamura et al. [4] developed a home health monitoring system for elderly and disabled persons at home, enabling a fully automated signal measurement with personal identification. They constructed a set of rooms and installed measurement devices in the bed, bathtub and toilet. The system was tested by 10 younger, healthy subjects who stayed overnight in the rooms. Data for three physiological parameters, i.e. bed temperature monitoring, ECG monitoring during bathing and body and excreta weight scale, were collected automatically, and the reliability of each monitoring system was evaluated by the comparison study. For example, the monitored bed temperature was compared with video recordings during sleep in terms of time in bed and number of body movements overnight. They argued that reliable measurements were made non-invasively and without the subjects’ awareness using the proposed system.

As a proof of concept, Masuda et al. [27] developed a telecare system for remotely monitoring heart rate and respiration. The home-side system consisted of an air-filled mat, a measurement unit and a bridge unit handling connection. When a person lies on the mat, his/her heartbeat and respiratory movements cause perturbations in the air pressure of the mat, which can easily be acquired by a pressure sensor. Through an appropriate filtering process, both heart rate and respiratory rate could be estimated. The proposed system was tested in several practical home-visit rehabilitation cases to evaluate its potential. A physical therapist from a hospital and his three patients acted as subjects in the experiment. The proposed system showed its usefulness for both the therapist and the patient in planning and evaluating daily rehabilitation training. With the domain knowledge on health and physical therapy, they investigated how the system would work among patients, therapists and smart systems effectively. Emphasizing a set of tools for the measurement, LeBellego et al. [28] evaluated a patient’s general health status through the automatic processing of criteria on the patient's activity in the natural environment. For the experimental set-up, they divided a hospital suite into five areas, each matching an area of interest (bedroom, entry, living room, toilets and shower). Most of the sensors worked distantly from the patient (ambient sensors) and detected movements or displacements (door detectors, infrared sensors). Collected data were analyzed with software that could perform many kinds of statistical analysis over long periods. Meetings with the medical staff were held periodically to determine whether their hypothesis was true. Based on the results, they argued that the system could provide long-term information about a patient’s health status for three ADL criteria: mobility, elimination and personal hygiene. Most importantly, they noted several interpretation issues of the observation such as the “noise” associated with the large number of visitors.

Chan et al. [29] developed a multisensory home monitoring system for elderly people living alone to observe behaviour deviation as an indicator of abnormal events. The motor activity data (in bed, getting up, getting out, visiting the toilet) were analyzed from a statistical perspective to assess changes in occurrence, time and duration. A trial was carried out over an eight-month period with four elderly persons. The nursing staff regularly called on the participants and recorded ratings concerning them. The system results were then compared with nursing staff ratings. Good agreement was found between the system results and the nursing staff ratings. West et al. [30] introduced the concept of “anxiety” to represent the time a device can be left unattended. They explored how different interactions with devices could change the anxiety level for devices in a smart house laboratory in which each device was augmented with sensors to detect interaction by the occupant. When a device was turned on, its anxiety was zero but would rise over time if not attended by the occupant. Eventually, when it reached a defined threshold, some action should be taken. Anxiety was represented by a number of statistical models that were combined together to integrate interactions with hazardous devices (stove, fridge) as well as other passive devices (doors, chairs) to compute anxiety as it varied with time. Results showed that the anxiety model had produced meaningful results for scenarios in the kitchen.

In the system effective evaluation, three user studies [25,27–29] included the domain knowledge, and three user studies [27–29] evaluated the proposed systems with end users. In developing and evaluating healthy smart home systems, domain knowledge should be included to fulfil the goals of health and safety effectively. For example, consultation with physicians and related clinical persons was necessary in order to find criteria for an appropriate evaluation of the healthy smart systems [4]. Further, understanding of end users’ limitations and capabilities in the proposed system was important to optimize the functions of the system through the evaluation [25]. Regarding the “human factors” category, only one user study assesses the “fit” between users and the technology in addition to the effectiveness of the system [27]. Several research issues were raised through the review of user studies in the system effectiveness evaluation that needed to be addressed in developing healthy smart home systems. First, many researchers on smart home projects presented their findings and understanding based on pilot studies normally completed with a small sample size over a short period [4,24–27], but user studies with larger samples sizes would need to be performed over a longer period in order to validate the effectiveness of the systems. Second, monitoring systems were usually performed based on the assumption that the subject was living alone [24,26,28,29]; however, in reality, there would be visitors. Thus, distinguishing the monitored subject from visitors would be essential to avoid incorrect interpretation of the monitoring data. Visits would be part of the subject’s environment to provide essential information about the subject’s socialization ability [28]. Third, ethical and social issues regarding monitoring humans during private ADL should be solved for the acceptance of using the surveillance technology. The extent of advantage over harm for an individual while using the vision technology would need to be identified in order to decide whether it is acceptable [25,29]. Fourth, most of the systems have been designed to measure only one or two variables. However, both physical and mental reasons may be implied in the changes of the patient’s behaviour. Thus, several psychological and physiological variables would need to be monitored simultaneously and be interpreted carefully [28]. Fifth, real situations occasionally may not match the hypothesis about the monitored subject (e.g., someone may go to the bathroom to pick up a forgotten item instead of performing personal hygiene). To interpret the monitoring data correctly, specific rules should be provided to distinguish “expected” from unexpected activities [28].

User Experience Evaluation

Nisbet [31] examined the potential benefits of integrated systems with an emphasis on input devices and control techniques. He conducted several case studies targeting users of wheelchairs in three different conditions: discrete systems with distributed controls, discrete systems with integrated controls and fully integrated systems. By exploring three main issues, control characteristics of target devices, practicality and appropriateness, advantages and disadvantages of each system were identified. The case studies indicated that the most important issue concerned the matching of the input device to the capabilities of the user and the control characteristics of the target devices. Based on the belief that a smart system should be led by end users’ requirements, Cooper and Keating [32] performed a user needs survey with 56 people with disabilities and 29 people at a rehabilitation day centre through detailed interviews, in order to determine a user specification for an integrated home system for rehabilitation. The key finding was that the automation of the home was of lesser importance to the user group than expected. Further, they expressed the fear of social isolation caused by independent living through technology that would reduce the need for human care services such as home helps, meal on wheels, etc.

Rantz et al. [33] conducted an interdisciplinary research project by integrating TigerPlace, a specially designed independent living centre, into the MU campus and the Columbia community. This was an active collaboration between computer engineering, health informatics and nursing within an academic health science centre to support older adults’ independent life within their residence. By emphasizing the concept “ageing in place”, a flexible technology infrastructure was implemented to identify individual risk factors such as falls, mobility, physiological measurements and sensory impairments. They investigated three focus groups with 15 seniors in a care retirement community. The results illustrated that people were willing to have the technology installed in their homes if it is reliable, affordable and not intrusive.

In the user experience evaluation, nine user studies, except for three [31,34,40] included the domain knowledge; and eleven user studies, except for one [40], evaluated the proposed systems with end users. As expected, user studies emphasizing UX would try to integrate end users in the evaluation of the healthy smart systems based on the domain knowledge of homes, health and human cognition. As expected regarding human factors, all user studies considered end users’ interaction with the smart devices and technology from a user perspective. Several research issues were also raised through the review of user studies in the UX evaluation of healthy smart home systems. First, systems should be developed based on the needs of the end users, not on technological capabilities. Disabilities and individual needs could vary considerably; therefore, each case would require a specialized solution. Thus, a wide variety of input and output solutions and accessibility is necessary to meet the needs of the users [36]. Further, it is essential that intuitive user interfaces are designed, targeted to the task, because incoherent or complicated control techniques could be difficult to learn and hard to use. Thus, the most efficient control regime for each task should be identified. It should not be assumed that simply because a user can manage a particular control in one task domain, it should be used for all [31]. Second, in order to ensure a smart home system would reach its full potential for the support of its users, the end users’ feedback about how the system performs in actual use is essential [31,38]. Through the evaluation by end users, systems can be optimized to match user capabilities and the intended purpose [8]. Third, addressing the complex issues of ageing would require many researchers with multiple perspectives. For example, researchers engaged in evaluation work with people with dementia should be aware of the needs of people with dementia [8]. Accordingly, an active collaboration of interdisciplinary participants such as potential users, caregivers, clinicians and therapists, engineers and service providers would be needed in the development stages of systems [32,33]. Fourth, users have expressed a fear that social isolation could be caused by smart home systems, which has enhanced their independence. The impact of the systems on users' lives should be identified when they are applied. Further, a holistic approach to meeting users’ needs using the technology is necessary to identify both positive and negative effects of the smart technologies on UX [32]. Fifth, technical installations ought to be invisible or unobtrusive. Additionally, the identification and profile of the individual must always be protected. For example, it would be acceptable if the actual video could not be viewed by an operator and images should not be stored. Such ethical issues are of importance when lifestyle monitoring systems in smart homes are discussed [25,36].

Understanding of UX: Perception, Need and Sensibility

Research on smart service and technology has focused mostly on technical achievements, limited to “proof-of-concept” applications. Although some research has dealt with evaluation, the primary concern was validation of the systems for the proposed functionality based on human behaviours or responses. There is little evaluation concerned with UX, including users’ perceptions, and would need smart technologies from a cognitive point of view. Without understanding the potential users’ perception of the system in practice, the unexpected side or negative effects of the technologies on users’ lives could be over-looked. UX, as a combination of users’ sensibility, emotion and cognition, should be considered for a successful technology integration and adoption of health smart systems by end users.

Inclusion of Domain Knowledge: Interdisciplinary Research

To cope with the requirements of the elderly or disabled, the domain knowledge such as housing studies, healthcare and human cognition should be incorporated into the development and evaluation of healthy smart systems. Without an understanding of the current problems in the proposed application areas, it would be difficult to argue for the validation and benefits of the proposed systems. There should be significant interdisciplinary research efforts incorporating contributions from architecture, health and human factors in addition to engineering. The lack of related expertise can be a major barrier to the adoption of smart technologies and services into practice. To realize suitable services for assisting users in reality, multiple perspectives from many researchers in different disciplines are required as a user-centric research.

Consideration of Usability: Intuitive and Multimodal Interaction

Consideration of the usability of the system is essential to increase the acceptability of the systems [37]. To enhance the usability, users’ interaction with systems should be effective, direct and easy to use; otherwise the smart homes would become uncomfortable places, thus displeasing their occupants. Elderly people have more difficulties with assistive technologies and devices, and badly designed interfaces could be quickly abandoned [5]. An intuitively customized user interface appropriate to user needs would enable efficient interaction in controlling the technology as a transparent accessing method. Multi-modality is one of the promising notions for interfaces. Multi-modal user interfaces can support robustness by allowing users to choose the most appropriate modality for the interaction needs [31,35].

With the shift to an ageing society, the smart home is becoming indispensable. Technological support could play an important role in assisting elderly or disabled people to be independent in their own homes. It was noted that more consideration of human factors and domain knowledge are needed in healthy smart homes research in order to make them acceptable and adaptable for the end users. Accordingly, human-computer interaction (HCI) and cognitive aspects of human behaviour are currently emphasized in user studies for the development of smart technologies and devices. The knowledge and lessons that we have learned from the critical review of user studies can be sources of inspiration for future studies on smart homes. Further, valuable insights on cognitive paradigms of smart technologies could be derived which would contribute to the development of smart environments that ultimately support users’ satisfaction and sensibility.