Home Observation for Measurement of the Environment: Collecting Data Virtually

Abstract

This study examined the feasibility and the reliability of a virtual assessment of the home environment using the widely used measure of the home environment, the Home Observation for Measurement of the Environment (HOME) Inventory. A convenience sample of 53 families were assessed with the virtual and in-person HOME. Children were ages 6 to 36 months and lived in 2 states in the United States. When examining HOME dimensions, the mean level of agreement was not statistically different from 85% except for the Involvement dimension. After having received both an interview in-person and virtually, most parents reported preferring the virtual (63.3%) over the in-person interview (36.7%). Results from this study show that the HOME Inventory can be administered virtually for many families and that the data collected is useful for research and applied purposes.

Keywords

home environment virtual assessment family

What’s New

The widely used Home Observation for Measurement of the Environment (HOME) Inventory can be used in virtual assessment using existing videoconferencing technology.

Introduction

One of the most widely used measures of the home environment for both research on child development and structuring and evaluating the effectiveness of parenting interventions is the HOME Inventory.¹ The HOME was designed to be administered during a visit to a child’s home with both the child and key caregivers present. The instrument uses a combination of semi-structured interview and direct observation data collection. During administration of HOME, the interviewer converses with the child’s primary caregiver and observes the physical characteristics of the setting and interactions between the caregiver and child. While the HOME was designed for in-person administration, successfully adapting the HOME for virtual administration could reduce administration costs and increase accessibility.

Here, we examined whether virtual administration of the HOME would result in similar scores as would appear for a family during the in-person administration. The in-person HOME was treated as the “gold standard” against which to evaluate the usefulness and validity of the virtual HOME administration. We explored accommodations needed for readily available video conferencing technology and the comfort of participants, consistent with approaches used in evaluating the quality of virtual health and human service visits.^2-5

Methods

Sample and Study Design

Study eligibility included English-speaking mothers of children ages 6 to 36 months with reliable Internet service. A convenience sample was recruited using social media, email invitations to participants in other studies of young children, and research pools at the University of Arkansas for Medical Sciences and Cincinnati Children’s Hospital Medical Center. The study was approved by the Duke University Institutional Review Board.

Participants were screened for eligibility and completed survey measures. Participants were then scheduled for HOME assessments, 1 virtual and 1 in person, both of which were recorded by study staff. The order of virtual and in-person sessions was counterbalanced by family and conducted with different interviewers. During the virtual administration, participants used their own devices for teleconferencing (eg, smartphones, tablets). Consistent with recommendations in the administration manual for the HOME,¹ the visits were intended to last at least 30 minutes, thus allowing time to make the observations needed to score items and time for comfortable conversational exchanges. Following the final HOME assessment, interviewers asked participants about their experiences and rated issues related to both virtual and in-person HOME assessment experiences.

Measures

The Infant-Toddler version of the HOME includes 45 items designed to assess 6 dimensions of a child’s home environment. The dimension Responsivity measures caregiver responses of a supportive type to the child’s behavior, offering verbal, tactile, and emotional reinforcement, and communicating freely through words and actions (eg, “Parent’s voice conveys positive feelings toward child”). Acceptance measures parental acceptance of less desirable behavior from the child and the avoidance of excessive restriction and punishment (eg, “No more than 1 instance of physical punishment during past week”). The dimension Organization measures regularity and predictability in the family’s schedule and space and the use of community services as part of the family support system (“Childcare, if used, is provided by one of 3 regular substitutes”). The Learning Materials domain assesses the child’s access to developmentally stimulating play and learning materials. Involvement measures the caregiver’s active involvement in the child’s learning and provision of stimulation for increasingly mature behavior (eg, “Parent provides toys that challenge a child to develop new skills”). Finally, the domain Variety measures the breadth of experiences that are included in the child’s life (eg, “Child has 3 or more books of his/her own”).

Accommodation for Virtual Administration

The study team modified the HOME manual and training to account for differences in administration between the in-person and virtual administration. The study team examined HOME items to consider whether they may have been more difficult to observe in a virtual visit, specifically if the size and orientation of the device may have presented barriers to observation. For example, it may be more difficult to capture nuances of facial expression such as a parent’s annoyance toward the child (Item #15) during a virtual compared to an in-person HOME administration. Three things emerged from these considerations: (a) minor adjustments in the instructions for administering and scoring, (b) specification of the types of devices needed and how to guide caregivers in the use of those devices, and (c) training of interviewers on procedures to use for virtual visits. Modifications were developed in accordance with recommended best practices for virtual data collection,^2,4,6 which included ensuring that (1) technology allowed adequate observation of key materials and actions, (2) participants had adequate understanding of how to use the technology needed for the visit, (3) sufficient time to gather the information needed to accurately document key conditions, and (4) adequate support for redirecting the conversation and image capturing when issues arose during a visit. The team then performed 6 field tests during which information was gathered on difficulties with virtual data collection and adjusted in procedures. See Supplemental Materials Table 1 for details.

Training of interviewers included a review of the manual and 2 virtual training sessions attended by project leadership and interviewers. Interviewers practiced the protocol with one another. Each interviewer scored the same virtual HOME video completed by a gold standard coder. Interviewers were required to achieve a score of 85% agreement to be certified. A second video was submitted by each interviewer to the gold standard coder about halfway through the data collection period to monitor for drift.

Data Analysis

The length of HOME sessions across delivery time (in-person and virtual) was compared using a paired t-test. Scores on each HOME dimension were examined to determine agreement in ratings between the virtual and in-person visits.^7,8 Inter-rater reliability was assessed using the percent agreement, calculated using interval-by-interval agreement. Specifically, for each participant, the number of items with agreement was divided by the total number of items and multiplied by 100. These values were then averaged across participants. Confidence intervals for the percent agreement were calculated for each dimension using the Clopper-Pearson interval.⁹ While there is no single value for the percent agreement that indicates a nonconsequential disagreement, there is a general consensus that 75% to 90% demonstrates an acceptable level of agreement,^10,11 as 85% reliability is the standard for reliability training; we interpret that level of agreement as being roughly equivalent.

Results

Demographic characteristics are shown in Table 1. The sample (n = 53 mother-child dyads) was primarily white, and the mean age of mothers was 31 years. Most had at least some college and worked 30 hours or more a week. Most mothers lived with a partner. Family income was diverse, with almost a third of the sample earning less than $30 000 per year and slightly over a third earning more than $100 000 per year. Child participants averaged 17 months of age, and there were similar numbers of boys and girls.

Table 1.

Sample Characteristics (N = 53).

	Full sample
Parent age: years at child’s birth (M, SD)	31.3 (4)
Education
High school or less	18.8%
Some college or degree	81.3%
Employment status
Full-time	72.9%
Part-time	4.2%
Not working	22.9%
Partner living in home (Yes)	79.2%
Child age: months (M, SD)	17.1 (8.2)
Child is male	45.8%
Child race/ethnicity
White	69.6%
African American	15.2%
Hispanic	6.5%
Multi-racial/ethnic	8.7%
Family Income
Less than $30K	30.8%
$30K-$70K	9.6%
$70K-$100K	13.5%
More than $100K	36.5%
Unreported	9.6%

Abbreviations: M, mean; SD, standard deviation.

There were 53 paired home and virtual HOME assessments. Results of the counterbalance indicated the protocol was implemented successfully with 54% of participants first receiving an in-person HOME and 46% of participants first receiving a virtual HOME. The average length of time between the in-person and virtual HOME visits was 2 days (SD = 19.8), and 90.6% of the interview sessions were completed within 30 days. The median length of the in-person HOME was 40 minutes (M = 39.5, SD = 12.0), and the virtual HOME was 38.7 minutes (M = 38.6, SD = 12.6). More specifically, 11.8% of in-person assessments were less than 30 minutes and 28.8% of virtual HOMEs were less than 30 minutes, with no statistical difference, t(52) =.382, P =.70.

Scores on each HOME dimension showed high percent agreement (Table 2). The mean level of agreement was not statistically different from 85% for all HOME dimensions, except for Involvement (mean percent agreement = 77.4; 95% CI = [72.4-81.8]). In addition to examining dimension scores, each HOME item was also examined to determine agreement in scoring between virtual and in-person visits. Percent agreement was above 85% for 30 of the 45 HOME items, between 75% and 85% for 8 items, and under 75% for 7 HOME items (see Supplemental Materials). The Involvement dimension included the most items with less than 85% agreement (n = 4), while the Acceptance dimension included the fewest (n = 1).

Table 2.

HOME Dimensions Scores by Location of Assessment.

HOME dimension	Number of items	In-person HOMEmean (SD)	Virtual HOMEmean (SD)	Percent agreementmean (95% CI)
1. Responsivity	11	8.94 (1.15)	8.81 (1.52)	86.1 (83.0-88.8)
2. Acceptance	7	6.53 (0.61)	6.32 (0.80)	93.3 (90.2-95.6)
3. Organization	6	5.19 (0.90)	5.13 (0.81)	87.1 (82.9-90.6)
4. Stimulation	9	8.25 (0.96)	8.4 (0.82)	90.8 (87.8-93.2)
5. Involvement	6	5.02 (1.13)	4.83 (1.22)	77.4 (72.4-81.8)
6. Variety	6	4.68 (1.14)	4.79 (1.18)	86.8 (82.6-90.3)
Home Total	45	39.6 (3.64)	39.28 (3.89)	87.2 (85.8-88.5)

Abbreviations: SD, standard deviation; SE, standardized error.

Note. N = 53. Items scored 0/1. Dimensions scored as a sum of items. Higher scores indicate more enriched HOME. The Clopper-Pearson confidence intervals were calculated.

Of the 15 items with less than 85% agreement, 7 were collected by interview (eg, “Parent talks to child while doing household work” and “Family visits relatives or receives visits once a month or so”), 6 were collected by direct observation (eg, “Parent tells child name of object or person during visit”) and 2 were collected by either interview or direct observation (eg, “At least 10 books are present and visible”).

For 13 of the 15 items with less than 85% agreement, there was no evidence of bias using virtual data collection. We considered differences less than or equal to 10% as random variance. For example, “Parent caresses or kisses child at least once” was scored yes in the virtual interview and no in the in-person interview 13.2% of the time. By comparison, this same item was scored yes in-person and no during the virtual HOME 15.1% of the time. The difference (15.1%-13.2%) was 1.9%. However, “Complex eye-hand coordination toys,” which is collected either using interview questions or observation, was scored yes 10 times in the virtual interview compared to yes 3 times in the in-person interview, and “Parent invests in maturing toys with value via personal attention,” which is collected by interview, was scored yes 9 times in the in-home interview while only 3 times in the virtual interview.

Device caregivers used for virtual interviews were smart phones (71.2%), laptops (19.2%), tablets (3.8%), and a desktop computer (1.9%); 2 devices (3.8%) were not reported. Interviewers reported that technology complications did not interfere with 36 (67.9%) virtual visits, slightly or somewhat interfered with 16 visits (30.2%), and were very much a problem for 1 visit. Internet interruptions were recorded during 3 visits. Interviewers rated camera use as “awkward” in less than 10% of visits (9.4%).

Most parents reported preferring the virtual (63.3%) over the in-person (36.7%) HOME administration. However, in follow-up comments, many parents indicated either would be acceptable. For example, a participant reported, “Yeah it was fine . . . I’m used to Zoom by now; it feels normal to me.” Other participants reported, “It was fine, feels the same as virtual. In-person was harder to schedule” and “Prefer in-person, tired of zoom meetings, but either is fine.”

Discussion

Providing efficacious virtual health and social services requires the collection of data to guide and evaluate practice. However, most measures were designed to be implemented in person. Therefore, it is critical to determine whether measures designed for in-person administration can yield valid information when used in a virtual format. The findings from this study indicate that valid data can be obtained through virtual administrations of the HOME Inventory.

When examining HOME dimensions, the mean level of agreement was not statistically different from 85%, except for the Involvement dimension. The Involvement dimension includes 6 items about how involved a parent is in a child’s daily life and developmental progress. Given that these items are primarily collected using interview questions (as opposed to observation), it is difficult to reconcile why the questions would be asked and/or answered differently in a virtual vs in-person administration. However, multiple items on this scale capture developmental scaffolding (eg, investing in maturing toys with personal attention, structuring the child’s play), making it possible that the exemplars chosen during the administrations varied across developmental domain and the interviewers did not further inquire about other areas of development. As such, the manual and training would benefit from additional modifications, providing scripted interview questions to ensure reliable parent reporting.

While there was concern that there would be discrepancies in scoring of HOME items that involve observation as the mode of data collection given potential difficulties in viewing all relevant interpersonal exchanges and household physical conditions via phones and other media platforms, our results do not seem to indicate challenges for observation during virtual implementation. Moreover, there was not an obvious systematic bias in the scores for items that use observation as the mode of data collection.

Experiences with the virtual HOME implementation were primarily positive. Both parents and interviewers found the virtual administration to be without major technology barriers, although the sample of families included was primarily higher income. When asked to compare the in-person and virtual HOME, parents preferred the virtual administration. This is not an uncommon preference, eg, families in home-visiting programs also reported their appreciation of the flexibility that virtual visits provide.¹²

Limitations

Although interviewers were able to follow protocol for administering the 53 virtual and in-person visits used to score HOME, the study has some limitations. The sample size was modest (N = 53) and families in the convenience sample were primarily of middle and upper income. A recent study documented that while digital access in lower-income families is relatively prevalent (nearly 70%), families without digital access (eg, those without broadband home Internet or a smartphone with a data plan) had higher odds of being lower-income.¹³ It would be useful for future studies to include a sample with greater economic diversity to better understand if there are additional barriers or concerns about the virtual HOME that were not captured with the current sample. Additionally, there were not enough cases using each of the forms of data collection (eg, smartphones, laptops, desktops) to make a comparison of the advantages or disadvantages of types of devices used.

Very few of the visits involved members of the family other than the target child and the primary caregiver. Therefore, it is unclear whether there might be more discrepancies in between administration types (virtual vs in-person) for families where other adults play a strong role in caregiving or where other children may change the behavior of the target child or primary caregiver. It would also be useful for future studies to collect direct measures of children’s development to confirm whether scores across data collection modality share the same associations with intended outcomes.

Conclusions and Implications

Results from this study show that the HOME can be administered virtually for many families and that the data collected can be useful for research and applied purposes. The scores obtained from virtual administration were like those obtained using the standard in-person approach with little sign of systematic bias in the scores. However, it will be important to revise the HOME manual so that it more specifically addresses how to collect items for which there was less agreement between data collection modalities. Being able to collect key information about parents, parent-child interactions, and household conditions virtually would be particularly advantageous for those delivering home-based parenting support programs and other health and human services or conducting research when home visits are difficult to achieve (eg, for families in rural and frontier geographic areas).^5,14-18

Author Contributions

Dr Lorraine M. McKelvey conceptualized and designed the study and drafted and revised the manuscript. Dr Robert H. Bradley conceptualized and designed the study and drafted and revised the manuscript. Dr Leanne Whiteside-Mansell conceptualized and designed the study, designed the data collection instruments, supervised the collected data, conducted analyses, and drafted the initial manuscript. Dr Whiteside-Mansell passed away as of the submission of this manuscript. Drs Ashley Acheson and Xiawei Ou conceptualized the study, critically reviewed, and revised the manuscript. Drs Stephanie Merhar and Jennifer Vannest supervised data collection and reviewed and revised the initial manuscript. Drs Weili Lin and Karen M Grewen supervised data collection and reviewed and revised the manuscript. Dr Kanna Lewis conducted analyses and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Supplemental Material

sj-docx-1-cpj-10.1177_00099228261455066 – Supplemental material for Home Observation for Measurement of the Environment: Collecting Data Virtually

Supplemental material, sj-docx-1-cpj-10.1177_00099228261455066 for Home Observation for Measurement of the Environment: Collecting Data Virtually by Lorraine M. McKelvey, Robert H. Bradley, Leanne Whiteside-Mansell, Ashley Acheson, Xiawei Ou, Stephanie Merhar, Jennifer Vannest, Weili Lin, Karen M. Grewen and Kanna N. Lewis in Clinical Pediatrics

Supplemental Material

sj-docx-2-cpj-10.1177_00099228261455066 – Supplemental material for Home Observation for Measurement of the Environment: Collecting Data Virtually

Supplemental material, sj-docx-2-cpj-10.1177_00099228261455066 for Home Observation for Measurement of the Environment: Collecting Data Virtually by Lorraine M. McKelvey, Robert H. Bradley, Leanne Whiteside-Mansell, Ashley Acheson, Xiawei Ou, Stephanie Merhar, Jennifer Vannest, Weili Lin, Karen M. Grewen and Kanna N. Lewis in Clinical Pediatrics

Footnotes

ORCID iDs

Lorraine M. McKelvey

Leanne Whiteside-Mansell

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institutes of Health (grant numbers R34DA050261, R34DA050261-01S1, R34DA050261-01S2, R34DA050262, R34DA050262-01S1, R34DA050262-01S2, R34DA050268, R34DA050268-01S1, and R34DA050268-01S2.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: As the developer of HOME, Dr. Bradley and Arizona State University have a financial interest in the technology discussed in this publication. Dr. Bradley was not involved in sample recruitment or in data analyses. The remaining authors have no conflicts of interest relevant to this article to disclose.

Human Subjects

This study was formally reviewed and approved by the Duke University Institutional Review Board. All participants consented to participate and were compensated financially for their participation in the study.

Supplemental Material

Supplemental material for this article is available online.

References

Caldwell

Bradley

RH.

Home Observation for Measurement of the Environment: Administration Manual. The Connect Center, Arizona State University; 2016.

American Medical Association. Telehealth implementation playbook, 2022. Accessed August 29, 2022. https://www.ama-assn.org/system/files/ama-telehealth-playbook.pdf

Bock

Kakavand

Careaga

Gozalians

Shifting from in-person to virtual home visiting in Los Angeles county: impact on programmatic outcomes. Matern Child Health J. 2021;25(7):1025-1030.

Joint Task Force for the Development of Telepsychology Guidelines for Psychologists. Guidelines for the practice of telepsychology. Am Psychol. 2013;68(9):791-800.

Waters

Benton

Tschntz

Vandenberg

Winston

Virtual human service delivery: scan of implementation and lessons learned. Technology considerations for virtual human services delivery. ASPE, Office of the Assistant Secretary for Planning and Evaluation; 2021.

American Telemedicine Association. Core operational guidelines for telehealth services involving patient-provider interactions. Published May 2014. Accessed August 29, 2022. https://www.americantelemed.org/resources/core-operational-guidelines-for-telehealth-services-involving-provider-patient-interactions/

Kazdin

AE.

Research Design in Clinical Psychology. 6th ed. Cambridge University Press; 2023. doi:10.1017/9781108976589

Kennedy

CH.

Single-Case Designs for Applied Research. Sage; 2025. doi:10.4135/9781071916025

Clopper

Pearson

ES.

The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26(4):404-413. doi:10.2307/2331986

10.

Stemler

SE.

A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research, and Evaluation. 2004;9:1-11. doi:10.7275/96jp-xz07

11.

Graham

Milanowski

Miller

Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. Center for Educator Compensation Reform; 2012.

12.

Chazan-Cohen

Fisk

Ginsberg

, et al. Parents’ Experiences With Remote Home Visiting and Infant Mental Health Programs During COVID-19: Important Lessons for Future Service Delivery. Perigee Fund; 2021.

13.

Curtis

Clingan

Guo

Zhu

Mooney

Hser

YI.

Disparities in digital access among American rural and urban households and implications for telemedicine-based services. J Rural Health. 2022;38(3):512-518. doi:10.1111/jrh.12614

14.

Araujo

Dormal

Grantham-McGregor

Lazarte

Rubio-Codina

Schady

Home visiting at scale and child development. J Public Eco Plus. 2021;2:100003.

15.

Breton

Sullivan

Deville-Stoetzel

, et al. Telehealth challenges during COVID-19 as reported by primary healthcare physicians in Quebec and Massachusetts. BMC Fam Pract. 2021;22:192.

16.

Gajarawala

Pelkowski

JN.

Telehealth benefits and barriers. J Nurse Pract. 2021;17:218-221.

17.

Lefever

JEB

Bigelow

Carta

, et al. Long-term impact of a cell phone-enhanced parenting intervention. Child Maltreat. 2017;22(4):305-314.

18.

Mohammed

Hyuseni

Bui

, et al. Exploring the use and challenges of implementing virtual visits during COVID-10 in primary care and lessons for sustained use. PLoS ONE. 2021;16(6):e0253665.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.02 MB