Health App Review Tool: Matching mobile apps to Alzheimer’s populations (HART Match)

Development

The development of the HART was informed by existing mHealth assessment measures, literature on AD and AD caregiver population needs, clinical knowledge, and stakeholder data collected throughout focus groups. The HART was designed as a two-component assessment measuring (1) various usability and usefulness factors of apps, and (2) characteristics and ability considerations of AD/AD-caregiver users. Questions were designed to be answered by clinicians or healthcare providers who would be familiar with the general health factors, needs, and abilities of AD/AD-caregiver users. HART assessment items were designed to encompass the stakeholder perspectives, and were informed by existing mHealth and computer system measures.^5–9

Preliminary focus groups were held to assess the stakeholder perspectives. Stakeholder groups of interest included both individuals with AD, their primary caregivers, and healthcare professionals. HART items were categorized according to themes extracted from both AD population and healthcare professional focus groups to include the following: usability, external use factors, specific needs, functions, usefulness, interaction and usage of app, adoption of app, external variables, and internal barriers (theme definitions and supporting data to be reported elsewhere) and additional items were added as needed to ensure that each theme was holistically addressed. Once all themes were sufficiently developed, the items for the app and user assessments were categorized into domains. The HART was then compared to pre-existing measures and literature. In sum, the HART was developed using a multi-step method that incorporated structuring and re-structuring HART content to maximize its ability to measure relevant variables, its user-centeredness, and usability.

Once a complete list of questions was developed for the HART measure, questions were organized into two components, the App Assessment and User Assessment. App assessment items cover variables such as the function of the app, evidence base of the app (if relevant), complexity of the app, cognitive burden of the app, and accessibility of the app. User assessment items cover topics such as the health and function needs of the intended user, their technology experience, technology support availability, education level, disease severity of the individual with AD, and physical accessibility needs. Rating scale categories between app items and user items were complementary. For example, items on the technological savvy of a client were rated on a 5 point scale ranging from low to high technological ability, while items on the complexity of the app were rated on a 5 point scale ranging from low to high complexity; these items could then be compared to determine if the technological savvy of the client was appropriately suited for the complexity of the app. Complementary App and User assessment domains are ultimately matched to across the following categories: (1) usefulness (will the app meet user specific needs), (2) complexity (considers app complexity and user preparedness to accommodate this complexity), (3) accessibility, and (4) external use factors. The final version of the HART included 106 items; the App assessment piece consisted of items 1–59, and the remaining 60–106 items formed the User assessment.

Scoring

The HART scoring system included a series of Likert scale questions spanning Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree; and dichotomous Yes–No response options. These scores were translated into 1–5 scale for Likert scores and either 0 or 4 for most dichotomous responses. Differences between comparable items on each assessment component can then be compared. For example: Auditory Ability Score of 4 − Auditory Accessibility Score of 1 = Difference Score 3. The difference between comparable items was taken to determine the level of “match.” This resulted in five categories, Perfect fit, Good fit, Fair fit, Poor fit, and Very Poor fit with a difference of zero, one, two, three, and four, respectively.

Some of the comparable items in the App and User assessments could be assessed through a simple subtraction equation to attain the difference between item scores while others required more complex calculations. For example, several items are rated such that a higher rating indicates that the app is more suited to accommodate those with poor technology abilities; however, the comparable user items are rated such that higher scores indicate higher technology capacity. In such cases, one of these scores would need to be reverse coded prior to calculating the difference. This specific example also poses an additional problem—greater technological ability should not lead to a “Poor” or “Very Poor” fit rating. An app’s accessible design may very well make it particularly suited for a user with greater needs, but it would not also make it poorly suited for a user with lesser needs. Instead, a more universal design approach was taken such that an app with high support in an area of great need yields a high score, while this same app would still yield at least a good score in the case of a user who has lower need, e.g. greater ability. To accomplish this, scoring for some items was both reversed and truncated so that they could not receive a score lower than three and the difference between the complementary item would then not exceed two (“Good” fit). In other words, the difference would never yield a “Poor” or “Very Poor” fit. This in essence created a neutral option.

All scoring calculations were designed as an Excel function so that a detailed scoring key could be developed and assessed. Calculations were then translated to SAS syntax to ensure that scoring could be completed in a reliable manner.

Limitations

The novel nature of the HART’s design can be viewed as a limitation in that it deviates from typical assessment measurement approaches. However, the increasing focus on individualized care propels us to develop better ways of assessing the match between individual needs and potential solutions. Therefore, we anticipate that the HART will not only be effective within the intended population but will serve as a model for future matching tools specific to other diagnostic populations.

Future directions

The HART was created to be both client centered and structurally sound. Future research should evaluate the HART through formal psychometric assessments. Specifically, confirmatory factor analysis should be used to assess the unidimensionality and factor loading of the HART. In addition, Rasch analysis should be used to assess the reliability, rating scale structure, and the fit of intended users to the HART.