The Effect of Alternative E-mail Contact Timing Strategies on Response Rates in a Self-administered Web Survey

Abstract

The Federal Employee Viewpoint Survey is an annual survey of over 800,000 permanently employed civilian personnel from 87 agencies. First administered in 2002, the web-based survey measures a broad range of employee perceptions, attitudes, and behaviors, serving as a valuable tool for human resources managers to determine which aspects of an organization are working well and which may require intervention. The data collection protocol begins by sending all sampled individuals an initial invitation to participate. Thereafter, nonrespondents are sent weekly reminder e-mails. These notifications are typically sent on Tuesday mornings. In this article, we present results from an experiment investigating two alternative protocols varying when survey notification and reminder e-mails are sent. Interestingly, the stable Tuesday morning strategy produced a significantly higher response rate than the two alternatives, and we analyze timestamp information from one of the alternatives to help provide insight as to why.

Background

The U.S. Office of Personnel Management (OPM) conducts the Federal Employee Viewpoint Survey (FEVS; www.fedview.opm.gov) to measure how effectively agencies are managing their most important asset, their workforce. The web-based survey is sent to sampled individuals via a personalized link embedded in an e-mail message. The instrument taps a broad range of employee perceptions and behaviors proven to drive satisfaction, commitment, engagement, productivity, and retention. Originally called the Federal Human Capital Survey, the survey was first administered in 2002 and biennially thereafter. In 2010, it was renamed as FEVS and became an annual survey. As the solid blue line in Figure 1 illustrates, the sample size has increased appreciably since that time, the result of a progressively more intricate stratification scheme needed to meet the participating agencies’ demand for analyzing work units deeper and deeper within the organization. This increase culminated in a full census¹ of the federal workforce in FEVS 2012, although sampling resumed in FEVS 2013. More details on the current sampling methodology can be found in the survey’s technical report (U.S. OPM 2014).

Figure 1.

Sample sizes and response rates in the Federal Employee Viewpoint Survey, 2002–2014. Response rates in this figure are unweighted, but plotting the base-weighted response rates, which takes into account changes in the sample composition over time, shows a similar downward trend.

A spate of research over the last 15 or so years has found that response rates to surveys in the United States and throughout the world have been declining (Atrostic et al. 2001; Brick and Williams 2013; Curtin et al. 2005; de Leeuw and de Heer 2002; Petroni et al. 2004). As illustrated by the dashed red line in Figure 1, the FEVS is no exception to that trend. A response rate greater than 50% was once routinely attainable, but that threshold has not been reached since FEVS 2010. The lowest response rate on record was that of the FEVS 2012 census (46.1%), and the most recent FEVS administration’s response rate hardly surpassed that mark (46.8%).

The current data collection procedure for the FEVS begins by assigning agencies to one of the two cohorts that are deployed into the field one week apart in late April or early May. Table 1 summarizes the e-mail contact sequence for an example agency in the first launch cohort of FEVS 2014. Each agency, regardless of cohort, is allotted a six-week field period. In the first week, an initial e-mail invitation is sent to all sampled employees, typically on a Tuesday. Nonrespondents are sent weekly reminder e-mails, generally on the same day and time. One exception, however, is that the final reminder is sent on the morning of the last Friday in the field period, alerting those who have yet to respond that the survey will close at the end of the day. No monetary or token incentives are offered, but employees are notified that they are permitted to complete the survey during work hours.

Table 1.

Federal Employee Viewpoint Survey (2014) Invitation and Reminder Schedule for an Example Agency in the First Launch Cohort.

Field Period Week	Date	Day of Week	E-mail Type
1	April 29	Tuesday	Initial invitation to participate
2	May 6	Tuesday	Reminder #1
3	May 13	Tuesday	Reminder #2
4	May 20	Tuesday	Reminder #3
5	May 27	Tuesday	Reminder #4
6	June 3	Tuesday	Reminder #5
6	June 6	Friday	Reminder #6 (final—survey closes today)

While the current protocol fosters logistical tractability, a very welcome feature considering the cumbersome task of sending out roughly 5 million e-mails invitations and reminders over the course of the FEVS field period, one potential disadvantage is that a particular agency’s assigned contact time block may be inopportune for some of its employees, perhaps because of competing activities such as a standing meeting or other official duties rendering them less attuned to e-mail traffic. This is ostensibly an important factor, since approximately 60% of all FEVS 2014 completes were obtained on the same day an e-mail solicitation was sent. Bearing this in mind, we sought to determine whether imparting some variability into when e-mail reminders are sent could ultimately improve the likelihood an employee responds to the survey and reverse the downward trend of response rates witnessed in recent FEVS administrations.

The survey methodology literature abounds with articles proposing and evaluating various contact timing strategies. Many reside within the milieu of interviewer-administered surveys, whether conducted by telephone (Brick et al. 1996; Greenberg and Stokes 1990; Weeks et al. 1987) or face-to-face in a cross-sectional (Purdon et al. 1999; Wagner 2013) or panel survey context (Durrant et al. 2013; Lipps 2012), but an emerging strand of literature has begun to evaluate the effect alternative e-mail contact strategies on response rates self-administered web-based surveys. Results have been mixed. Several studies were unable to report any substantive differences (Sauermann and Roach 2013; Shinn et al. 2007; Sinibaldi and Hansen 2008; Yun and Trumbo 2000); however, based on a meta-analysis of employee satisfaction surveys administered via SurveyMonkey® (www.surveymonkey.com) in 2009 and 2010, Zheng (2011) reports that response rates were highest for survey invitations sent out on Mondays. Faught et al. (2004), on the other hand, found that sending e-mails exclusively on Wednesday mornings yielded the highest response rate in an establishment survey, which is also a strategy Quinn (2010) advocates in marketing research. Moreover, Bennett-Harper et al. (2007) and Paraschiv (2013) found Fridays to be superior. Although it seems unlikely a single e-mail contact protocol would work best for all target populations and survey topics, in light of the mixed results studies such as these have produced, we would argue more research into optimal e-mail contact strategies in self-administered web-based surveys is warranted. This article serves as one contribution in that regard.

Data and Methods

The goal of our experiment was to investigate whether, and to what extent, an alternative e-mail contact sequence could improve response rates in the FEVS. As such, we did not endeavor to conduct the experiment on all 87 agencies participating in the FEVS 2015. Instead, we restricted our focus to one large agency, the U.S. Department of Defense (DoD) Fourth Estate. In a nutshell, DoD Fourth Estate is comprised of civilian DoD personnel not directly employed as part of the U.S. Army, U.S. Navy, or U.S. Air Force. Components of the agency include the Office of the Secretary of Defense, Defense Information Systems Agency, Defense Contract Audit Agency, Missile Defense Agency, and Defense Commissary Agency to name a few. We felt this was a fitting agency on which to concentrate our efforts because it has historically achieved below-average FEVS response rates, and survey practitioners at the agency have demonstrated a keen interest over the years in researching the causes, correlates, and potentially detrimental impact of nonresponse in other surveys sponsored by the agency (e.g., Caplan and Hoover 2008; Caplan and Quigley 2005; Caplan et al. 2004).

The sampling frame for the FEVS is derived from an expansive personnel database maintained by OPM called the Statistical Data Mart of the Enterprise Human Resources Integration (EHRI-SDM). For more details on its contents and scope, see http://www.fedscope.opm.gov/datadefn/aehri_sdm.asp. Following the single-stage stratified sample design described in the U.S. OPM (2014), a total of 34,799 DoD Fourth Estate employees were selected for participation in the 2015 FEVS. The data collection period began on Tuesday, April 28, and ended on Friday, June 5. Prior to the start of the field period, sampled employees were randomly assigned to one of the three experimental groups of approximately equal size. The first group received the traditional FEVS e-mail contact sequence with the initial invitation and all subsequent reminders sent on Tuesday morning. The lone exception was that a final reminder was sent on the morning of Friday, June 5, the day the survey closed. All nonresponding employees received this final reminder regardless of experimental group.

The second experimental group received what we refer to as a rotating cohort strategy. Each employee was first placed at random into one of the six cohorts, which in turn was assigned to one of the following six time blocks for purposes of the initial e-mail invitation: (1) Tuesday a.m., (2) Tuesday p.m., (3) Wednesday a.m., (4) Wednesday p.m., (5) Thursday a.m., or (6) Thursday p.m. Each cohort was then rotated to the next time block for the ensuing week’s reminder. Figure 2 is provided to help visualize the procedure. For example, individuals assigned to cohort A received the initial e-mail solicitation on Tuesday morning, received the first reminder on Tuesday afternoon, and received the second reminder on Wednesday morning, and so on. Practical experience from prior FEVS administrations has suggested that fewer employees are on duty on Mondays and Fridays relative to the other three days in the work week, as evidenced by higher rates of out-of-office automatic e-mail replies, so those two days were excluded from consideration in this experiment.

Figure 2.

Illustration of the rotating cohort e-mail contact timing strategy.

E-mails sent during the morning time blocks went out at approximately 8 a.m. Eastern Standard Time, and e-mails sent during the afternoon time blocks went out at approximately 2 p.m. Eastern Standard Time. The bulk e-mailing software e-Campaign (version 10) (http://www.lmhsoft.com/ecamp/) was used to send out the e-mail invitations and reminders en masse. Note that because there were a total of six e-mail solicitations, all sampled employees were eligible to receive an e-mail during each of the six possible time blocks.

Bethlehem et al. (2011) define a typology of survey design features for employing principles of a responsive survey design as originally discussed in Groves and Heeringa (2006). The technique depicted in Figure 2 is what Bethlehem et al. (2011) would label a static adaptive design, since the information determining any sort of change in the contact protocol is known in advance of the data collection period. By contrast, Bethlehem et al. (2011) define a dynamic adaptive design to be one where protocol changes occur in response to observations made during the data collection period. Because the second alternative contact sequence we investigated fits within this paradigm, we refer to it as the dynamic adaptive strategy.

The dynamic adaptive strategy proceeded as follows. As with the rotating cohort strategy, employees were randomly allocated into one of the same six time blocks for purposes of the initial e-mail invitation. But rather than cycling employees through each time block in a predetermined fashion, auxiliary information on the sampling frame was used to fit a multinomial logistic regression model (Hosmer et al. 2013), where the dependent variable was the act of responding during one of the six time blocks. For modeling purposes, responses obtained on Friday or Saturday were treated as having occurred during the Thursday p.m. time block, while responses obtained on Sunday or Monday were treated as having occurred during the Tuesday a.m. time block. Predictor variables for the model included the employees’ agency component, supervisory status (nonsupervisor, supervisor, and executive), gender, an indicator of working part time as opposed to full time, an indicator of being a minority race/ethnicity, pay level, and an indicator of whether the individual’s job title falls within the umbrella of a science, technology, engineering, or mathematics (STEM) occupation. The model was (re)fitted at the conclusion of each week of the field period based on the cumulative data. The key statistic extracted was each nonresponding employee’s vector of time block–specific response propensities. The ensuing week’s reminder time block for these individuals was assigned stochastically in proportion to these propensities. Hence, the notion was to predispose nonrespondents to receive reminders during times which real-time evidence was suggesting comparable employees were responding at higher rates.

Results

Table 2 summarizes the sample sizes and response rates for each e-mail timing strategy. The response rates we report correspond to the RR1 formula of the American Association of Public Opinion Research (AAPOR) (2015). Because of an approximate six-month lag between the most recently available EHRI-SDM data used to develop the sampling frame and when the survey went into the field, there is some degree of ineligibility caused by employees who since departed the agency. These departures were identified by either out-of-office replies, ad hoc information provided by the agency, or via a systematic procedure whereby we verified future employment by means of a more up-to-date personnel roster derived from EHRI-SDM. Cases of known ineligibility were removed from the denominator of the response rate calculation as reflected by the line in Table 2 labeled “Adjusted Sample Size.” To be defined as a respondent, the individual must have answered at least 21 of the 84 core, nondemographic survey items. As documented in the U.S. OPM (2014), this is the conventional FEVS threshold for defining a complete case.

Table 2.

Sample Sizes and Response Rates by E-mail Timing Strategy.

	E-mail Timing Experimental Group
Measure	Traditional	Rotating Cohort	Dynamic Adaptive	Total
Initial sample size	11,599	11,600	11,600	34,799
Ineligibles	824	902	844	2,570
Adjusted sample size	10,775	10,698	10,756	32,229
Respondents	5,216	4,968	5,065	15,249
Response rate	48.4%	46.4%	47.1%	47.3%

Interestingly, the Tuesday morning e-mail contact strategy outperformed the two alternatives. The Tuesday morning approach yielded a response rate of 48.4%, whereas the rotating cohort strategy was 2 percentage points below that mark at 46.4%. At 47.1%, the dynamic adaptive approach was slightly better than the rotating cohort alternative, but it still lagged the traditional method. Although modest in magnitude, these differences are statistically significant (χ² = 8.68; p = .013).

We anticipated unveiling pockets of deviations from this overall finding as we inspected response rate differences across a range of domains—i.e., those defined by particular demographic categorizations or components of the agency—but the general pattern almost always held. Figure 3 illustrates this tendency based on a few example demographics: employee gender, whether a minority or nonminority race/ethnicity, supervisory status, and pay level. The traditional method generally garnered the highest response rate, followed by the dynamic adaptive and rotating cohort methods, respectively. There are a few instances where the dynamic adaptive method outperformed the rotating cohort methods, but both still lagged behind the Tuesday morning approach. One exception is that employees in the highest-pay level responded at the highest rate to the rotating cohort, but fewer than 3% of DoD Fourth Estate employees fall within this demographic category, so we caution that the finding could be simply attributable to sampling error rather than anything substantive.

Figure 3.

Response rates by e-mail timing strategy for select employee demographics.

Although our experimental design renders us unable to conclude that sending e-mails exclusively on Tuesday mornings would generate higher-response rates than, say, an e-mail invitation and reminder schedule restricted to Wednesday or Thursday mornings, deeper analyses into the rotating cohort design help paint a somewhat clearer picture of the Tuesday morning effect. By conditioning on the respondents in this experimental group and examining their distribution of ultimate reminders—i.e., the last reminder the employee received prior to completing the survey—we can assess which reminder was most influential. This is only possible because each employee was randomly assigned to one of the six possible time blocks for purposes of the initial invitation, and each was eligible to receive a reminder during all five remaining time blocks.

Figure 4 is a histogram of ultimate reminders for those responding as part of the rotating cohort experimental group. For purposes of this analysis, responses obtained after the last Friday reminder went out were linked back to the time block of the reminder sent earlier in that final field period week. The figure illustrates how the Tuesday morning e-mail was the most frequent lure. It also illustrates a subtle tendency for e-mails sent later in the week to be less effective. Another noteworthy point is that the marginal percentage of morning e-mails was slightly larger than that of afternoon e-mails—51.4% vs. 48.6%, respectively—which provides some evidence that, all else equal, sending e-mails in the morning appears to be more efficacious than sending e-mails in the afternoon.

Figure 4.

Time block distribution of reminders immediately preceding a response for the 4,968 respondents in the rotating cohort experimental group.

Concluding Remarks and Ideas for Further Research

Historically, the web-based FEVS has been governed by a data collection strategy whereby the initial invitation and all subsequent reminders to nonrespondents are sent during the same weekly time block, typically on Tuesday mornings. One exception is that a final reminder is sent on the Friday that the survey closes. In this article, we presented results from an experiment conducted on one agency participating in the 2015 FEVS with the aim of determining whether introducing some variability into the timing of e-mail reminders could increase response rates. Surprisingly, both alternatives we considered proved inferior to the conventional Tuesday morning approach. This finding prevailed across a range of domains defined by work units within the agency and employee demographics. Based on further analysis of the rotating cohort experimental group, one where e-mail time blocks were assigned randomly between Tuesday morning and Thursday afternoon, there is some modest evidence that sending e-mails during the Tuesday morning time block is most likely to trigger a response and that subsequent time blocks later on in the work week are gradually less likely. Based on these findings, the FEVS administration team has no plans to part with the e-mail invitation and reminder protocol currently in place.

The generalizability of our results is limited by the fact that we focused on only 1 of the 87 agencies that participate in the FEVS. Further research could investigate whether these findings are replicable across other agencies. In addition, as the FEVS is an organizational climate survey of a very specific population, employees of the U.S. federal government, there is no guarantee these results would extend directly to other survey topics and other target populations. Another factor that potentially limits the generalizability of our findings to other organizational climate surveys is that our experimental design excluded Mondays and Fridays. Although researchers have found Mondays (Zheng 2011) and Fridays (Bennett-Harper et al. 2007; Paraschiv 2013) to be effective days to send out e-mails in some survey settings, we were hesitant to do so for the present study because of the historically higher rates of out-of-office replies observed and in light of the fact that roughly 60% of 2014 FEVS responses were obtained on the same day the e-mail was sent. These concerns may be unfounded. A follow-up study could incorporate Mondays and Fridays into the regular sequence of e-mail invitations and reminders.

There are numerous other extensions to the present study worth pursuing. For example, future research could explore whether information about when the employee actually views the e-mail, perhaps captured by means of an e-mail read receipt, could be useful. This would enable us to make the important distinction between making contact and securing a response and could offer insight into whether certain times are associated with more or less lag time between e-mail receipt and survey completion. Another intriguing idea would be to utilize prior FEVS administration timestamp information. All but 16 of the agencies currently participating in the FEVS conduct a census of their workforce, so even with an overall response rate around 50%, there is likely considerable respondent overlap between two adjacent years’ administrations. Heeding a recommendation in Lipps (2012), Kreuter and Müller (2015) attempted to conduct interviews in a panel survey around the same time the individual had responded in a prior wave. They found that the technique helped improve contact rates but not response rates. It would be interesting to see how a method similar in spirit would perform in a self-administered web-based survey context.

Footnotes

Authors’ Note

The opinions, findings, and conclusions expressed in this article are those of the authors and do not necessarily reflect those of the U.S. Office of Personnel Management.

Acknowledgment

The authors would like to thank Bill Dristy of the U.S. Office of Personnel Management for his help sending out Federal Employee Viewpoint Survey invitations and reminders at the prescribed times.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

American Association of Public Opinion Research. 2015. Standard definitions: Final dispositions of case codes and outcome rates for surveys, 8th ed. Lenexa, KS: American Association of Public Opinion Research.

Atrostic

Bates

Burt

Silberstein

. 2001. Nonresponse in US government household surveys: Consistent measures, recent trends, and new insights. Journal of Official Statistics 17:209–26.

Bennett-Harper

O’Brien

Levin

David

Shipp

Campbell

Sienkewicz

. 2007. Boosting response rates: Are there optimal times to email respondents? Proceedings of the Biennial Conference of the European Survey Research Association, Prague, Czech Republic, June 25–29.

Bethlehem

Cobben

Schouten

. 2011. Handbook of nonresponse in household surveys. Hoboken, NJ: Wiley.

Brick

Allen

Cunningham

Maklan

. 1996. Outcomes of a calling protocol in a telephone survey. Proceedings of the Survey Research Methods Section, American Statistical Association, Joint Statistical Meetings, Chicago, IL, August 4–8.

Brick

Williams

. 2013. Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of Political and Social Science 645:36–59.

Caplan

Hoover

. 2008. Administrative methods of increasing response rates. Proceedings of the Federal Computer-Assisted Survey Information Collection (FedCASIC) Workshops, Washington, DC, March 11–13.

Caplan

Quigley

. 2005. Administrative interventions and their impact on response rates in military members’ attitude surveys. Proceedings of the Annual Conference of the American Association for Public Opinion Research, Miami, FL, May 12–15.

Caplan

Quigley

Mathew

. 2004. Nonresponse research in the Department of Defense. Proceedings of the Annual Conference of the American Association for Public Opinion Research, Phoenix, AZ, May 13–16.

10.

Curtin

Presser

Singer

. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69:87–98.

11.

de Leeuw

De Heer

. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey nonresponse, edited by Groves

Dillman

Eltinge

Little

, 41–54. New York: Wiley.

12.

Durrant

D’Arrigo

Müller

. 2013. Modeling call record data: Examples from cross-sectional and longitudinal surveys. In Improving surveys with paradata: Analytic uses of process information, edited by Kreuter

, 281–308. New York: Wiley.

13.

Faught

Whitten

Green

. 2004. Doing survey research on the Internet: Yes, timing does matter. Journal of Computer Information Systems 45:26–34.

14.

Greenberg

Stokes

. 1990. Developing an optimal call scheduling strategy for a telephone survey. Journal of Official Statistics 6:421–35.

15.

Groves

Heeringa

. 2006. Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistics Society, Series A, 169:439–57.

16.

Hosmer

Lemeshow

Sturdivant

. 2013. Applied logistic regression, 3rd ed. New York: Wiley.

17.

Kreuter

Müller

. 2015. A note on improving process efficiency in panel surveys with paradata. Field Methods 27:55–65.

18.

Lipps

2012. A note on improving contact times in panel surveys. Field Methods 24:95–111.

19.

Paraschiv

D.-C.

2013. Timing in a web based survey: An influential factor of the response rate. Proceedings of the International Conference Marketing—From Information to Decision, Cluj-Napoca, Romania, November 8–9.

20.

Petroni

Sigman

Willimack

Cohen

Tucker

. 2004. Response rates and nonresponse in establishment surveys—BLS and Census Bureau. https://www.bea.gov/about/pdf/ResponseratesnonresponseinestablishmentsurveysFESAC121404.pdf (accessed September 23, 2016).

21.

Purdon

Campanelli

Sturgis

. 1999. Interviewers’ calling strategies on face-to-face interview surveys. Journal of Official Statistics 15:199–216.

22.

Quinn

2010. Invite timing tips—What time of day and day of the week works? http://www.peoplepulse.com/resources/useful-articles/invite-timing-tips/ (accessed September 23, 2016).

23.

Sauermann

Roach

. 2013. Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Research Policy 42:273–86.

24.

Shinn

Baker

Briers

. 2007. Response patterns: Effect of day of receipt of an e-mailed survey instrument on response rate, response time, and response quality. Journal of Extension 45.

25.

Sinibaldi

Hansen

. 2008. When is the best time to invite a respondent? An analysis of e-mail invitation timing and response to a web survey. Proceedings of the Annual Conference of the American Association for Public Opinion Research, New Orleans, LA, May 15–18.

26.

U.S. OPM (U.S. Office of Personnel Management). 2014. 2014 Federal employee viewpoint survey: Technical report. http://www.fedview.opm.gov/2014/published/ (accessed September 23, 2016).

27.

Wagner

2013. Using paradata-driven models to improve contact rates in telephone and face-to-face surveys. In Improving surveys with paradata: Analytic uses of process information, edited by Kreuter

, 145–70. New York: Wiley.

28.

Weeks

Kulka

Pierson

. 1987. Optimal call scheduling for a telephone survey. Public Opinion Quarterly 51:540–49.

29.

Yun

Trumbo

. 2000. Comparative response to a survey executed by post, e-mail and web. Journal of Computer Mediated Communication 6:1–19.

30.

Zheng

2011. What day of the week should you send your survey? https://www.surveymonkey.com/blog/2011/08/16/day-of-the-week/ (accessed September 23, 2016).