Abstract
The American Family Health Study (AFHS) collected family health and fertility data from a national probability sample of persons aged 18–49 between September 2021 and May 2022, using web and mail exclusively. In July 2022, we surveyed AFHS respondents and gauged their willingness to become part of a national web panel that would create novel longitudinal data on these topics. We focus on predictors of willingness to participate, identifying the potential selection bias that this type of approach may introduce. We found that efforts of this type to create a national web panel may introduce potential selection bias in estimates based on the panel respondents, with individuals having higher socioeconomic status being more cooperative. Thus, alternative recruitment strategies and re-weighting of the subsample may be needed to further reduce selection bias. We present methodological implications of our results, limitations of our approach, and suggestions for further research on this topic.
Introduction
From 2020 to 2022, the American Family Health Study (AFHS) collected family health and fertility data from a national probability sample (split into two replicates) of persons aged 18–49 using web and mail modes of data collection. We asked AFHS respondents from the second replicate about their willingness to become part of a national web panel. Our substantive objective is to capture short-term dynamics in the intersection between social experiences and sexual and reproductive health behaviors. Initial tests of this approach in a single U.S. county demonstrate its value for the study of fertility and reproductive health (Guzzo et al. 2019), but no nationally representative data currently exist that provide short-term repeated measurements of important contraceptive use and reproductive health behaviors. The formation of a national probability-based web panel would provide demographic researchers with a new data source for advancing knowledge in these important substantive areas.
As the cost-per-unit of recruiting participants to a probability-based online panel has continued to rise in the past decade, recruiting panel members from an existing cross-sectional sample can be time saving, logistically convenient, and cost effective. However, this requires assessment of the potential selection bias that may be introduced by differential willingness to participate in follow-up surveys. The main aim of this study is to analyze predictors of AFHS respondents’ willingness to participate in the follow-up panel study, considering both survey responses on family health and fertility topics from the AFHS and other auxiliary variables that may serve as useful indicators of potential selection bias.
Methods
The American Family Health Study
The second national sample replicate of the AFHS was fielded between September 2021 and April 2022. The AFHS used a sequential mixed-mode web/mail protocol for household (HH) screening to identify eligible persons aged 18–49. One eligible individual was randomly selected from each HH. This individual was then invited (either by mail or email) to complete a 60-minute web survey, using a similar mixed-mode protocol that encouraged response via the web. Shorter paper questionnaires were offered to those who failed to respond to the initial survey requests while still encouraging them to complete the survey online. The AAPOR RR4 response rate for the second replicate was 11.1% (17.8% screening, 62.4% main survey). 1 Respondents were compensated $70 for completing the AFHS survey. Additional details regarding the AFHS screening, sample design and data collection methodology can be found elsewhere (see https://afhs.isr.umich.edu/about-the-study/afhs-methodology/; Wagner et al. 2022; West et al. n.d.). All respondents were given the same invitation protocol; that is, we did not experimentally vary the protocol. The full data collection protocol is outlined in Figure 1 of the Appendix.
Follow-up Panel Feasibility Study
The follow-up study was launched in July 2022, with the primary aim of gauging the willingness of AFHS respondents to respond to short surveys once a month for a total of 6 months. All respondents in the second replicate were invited to participate in the follow-up study and promised a $10 incentive for completing it. The follow-up survey took less than 10 minutes to complete and collected information on topics related to family planning, financial expectations, and mental well-being.
Outcome Variable
Our outcome variable for this analysis is a binary indicator (1 = responded to follow-up invitation AND indicated willingness to participate in a future panel study; 0 = either did not respond to the follow-up invitation OR responded to the follow-up invitation but did NOT indicate willingness to participate in a future panel study).
Predictor Variables
The AFHS data provide a wealth of information for predicting willingness to be part of the web panel and indicating the potential selection bias introduced by those who expressed willingness. We group these predictor variables into three categories: demographics, substantive outcomes, and auxiliary data. The AFHS fielded different questionnaires for male and female respondents, with some questions common to both questionnaires. Substantive outcomes used as predictors in our analysis were: (1) asked in both the male and female questionnaires, on the web and on paper; (2) not part of any branching questions so as to avoid having many missing values; and (3) considered key variables from the original National Survey of Family Growth (NSFG) on which the AFHS was based (https://www.cdc.gov/nchs/nsfg/nsfg_products.htm).
These criteria allowed us to assess selectivity among the follow-up study respondents expressing willingness and identify potential variables that could be used to adjust estimates for this selection bias. We examined pairwise correlations among these substantive variables and dropped any variables determined to be essentially redundant. We first treated missingness as a unique category on each variable, and then multiply imputed all missing responses as an alternative approach.
The AFHS also collected paradata and made use of selected auxiliary variables to improve sampling efficiency; we also used these variables in our predictive models. See Table 1 in the Appendix for complete descriptions of the predictor variables. Table 2 in the Appendix presents summary statistics for the outcome variables and predictor variables.
Results
In a multivariable logistic regression model, we found that the odds of expressing willingness to be part of the follow-up web panel study were significantly higher among respondents with the following socio-demographic characteristics (see Table 3 in the Appendix): (1) individuals with college degrees and above, compared to those with some college and below (AOR = 1.33, 95% CI = 1.01–1.74); and (2) Non-Hispanic Whites compared to non-Whites (AOR = 1.41, 95% CI = 1.0–1.84).
The adjusted odds of expressed willingness were also significantly higher for respondents with health insurance (AOR = 1.59, 95% CI = 1.07–2.35).
Regarding the auxiliary variables, a larger time gap between the main survey completion date and the follow-up study invitation date was negatively associated with willingness to participate in the follow up panel study. Respondents receiving non-response follow-up calls for the main survey had marginally lower odds of willingness (AOR = 0.59, 95% CI = 0.30–1.13). Those consenting to receive text messages had significantly higher odds of willingness (AOR = 1.85, 95% CI = 1.38–2.49). The ROC curve analysis indicates that this logistic model fit the data reasonably well (Area under ROC curve = 0.7). Supplemental multiple imputation analyses led to the same overall conclusions (see Table 4 in the online Appendix).
Table 5 in the Appendix presents sample demographic distributions by age group (18–30, 31–40, 41–49), gender (Female, Male), and race/ethnicity (Non-White, White) for respondents who completed the follow-up panel feasibility survey and expressed willingness to participate in a follow-up panel study (n = 838). We also provide corresponding population benchmark distributions obtained from 2020 ACS 5-Year Estimates Public Use Microdata Sample. This comparison reveals a slight mismatch (in terms of age, gender, and race/ethnicity cross-tabs) between the subsample of AFHS respondents expressing willingness and the underlying population.
Discussion
The results of this study suggest that efforts to create a national web panel by following up with initial respondents to a national cross-sectional web/mail survey may introduce potential selection bias in estimates based on the panel respondents. We found that respondents completing the follow-up panel feasibility survey and expressing willingness to participate in the panel had higher education, were more likely to be non-Hispanic Whites, had health insurance, and were willing to receive text messages related to the original study. Shorter periods of time between response to the initial survey and the follow-up invitation also increased the odds of expressed willingness. This suggests that panel recruitment of this form may result in more cooperative individuals with higher socioeconomic status, meaning that alternative recruitment strategies may be needed for other types of individuals. Our results also suggest the need to re-weight the resulting subsample (on top of the original weights) to make it more aligned with population benchmarks in terms of key demographic variables such as age, gender, and race/ethnicity.
This study was not without limitations that suggest directions for future work in this area. First, we did not investigate selection bias in the AFHS itself, but only the selection bias conditional on participating in AFHS. Some of these factors may be compounded if both are considered. Second, the relatively long lag between completion of the AFHS and the follow-up survey may have mattered. Ideally, respondents would be asked at the end of the AFHS interview about their willingness to participate in the panel. Third, we only looked at response to the follow-up survey (and stated willingness to participate in a panel) rather than actual participation in a panel. Future studies should analyze if stated willingness to participate leads to actual participation. Although our use of a yes/no scale for capturing behavioral intention is easy to understand, fast to administer, and requires minimum effort from our participants, it could miss nuanced aspects of willingness that may be captured using a more probabilistic (e.g., 0–100) scale. Despite these limitations, our study suggests that recruitment of a panel from an initial national web/mail survey is feasible, but potential selection bias needs careful evaluation.
Future research should also consider designing an experiment in which some respondents are recruited to become part of a panel “from scratch” (e.g., a general address-based panel), while some are recruited from an existing national web survey (like the AFHS), and then analyze if different types of individuals respond to different type of recruitment strategies. Future work should also investigate predictors of both responding to an initial survey AND responding to a follow-up study. Future work could also look at asking people about their interest immediately after the original study is completed to avoid the time lag issue as found in this current study.
Adaptive survey design procedures, considering different recruitment protocols for different strata of individuals based on the results of the current study (e.g., Wagner et al. 2022; West et al. n.d.; Zhang et al. 2023), may prove effective in future panel creation efforts of this type. For instance, panel recruitment efforts can concentrate on units that are less likely to respond by offering higher incentives, following up with a telephone call to non-responding units, and/or switching to a different survey mode. Such strategies, if effective, would reduce the need to use weights or other methods of statistical adjustment to correct panel estimates for this type of selection bias.
Supplemental Material
Supplemental Material - What Predicts Willingness to Participate in a Follow-Up Panel Study Among Respondents to a National Web/Mail Survey?
Supplemental Material for What Predicts Willingness to Participate in a Follow-Up Panel Study Among Respondents to a National Web/Mail Survey? by Htay-Wah Saw, Brady T. West, Mick P. Couper and William G. Axinn in Field Methods
Supplemental Material
Supplemental Material - What Predicts Willingness to Participate in a Follow-Up Panel Study Among Respondents to a National Web/Mail Survey?
Supplemental Material for What Predicts Willingness to Participate in a Follow-Up Panel Study Among Respondents to a National Web/Mail Survey? by Htay-Wah Saw, Brady T. West, Mick P. Couper and William G. Axinn in Field Methods
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) of the National Institutes of Health (grant number R01HD095920; PI: B.T. West; website: 
). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Supplemental Material
Supplemental material for this article is available online.
Note
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
