How Contextual Cues Impact Response and Conversion Rates of Online Surveys

Introduction

Marketers frequently use surveys to assess the effectiveness of marketing campaigns, including those in the tourism industry (Burke and Gitelson 1990; Pratt et al. 2010; Woodside 1990; Woodside and Dubelaar 2003). Using appropriate methodology, researchers can gauge the cost-per-visitor and return-on-investment of marketing campaigns (Pratt et al. 2010). With the wide adoption of the internet among consumers, especially in the developed countries, marketing and tourism researchers are conducting surveys online. In comparison to the paper-and-pencil surveys, online surveys are faster, easier, more flexible, and more cost-effective (Dillman 2007; Tierney 2000). However, many problems plague online surveys, including low response rates, poor representativeness, and biased results (Hwang and Fesenmaier 2004; Pan 2010). Thus, researchers need to investigate the validity of online survey results and the ways to increase response rates and to rectify the possible biases.

Some methods work well for improving survey response rates, such as precontacts, reminder messages, postcontacts, personalized solicitation messages, and university letterheads (Sheehan and McMillan 1999). However, response bias remains a problem with all surveys because of coverage errors and the self-selection of respondents (Best et al. 2001; Ross et al. 2003; Woodside and Dubelaar 2003). Biased results can also interact with response rates: low response rates due to self-selection when a survey identifies one specific brand may bloat the conversion rates (Woodside and Dubelaar 2003). In a meta-study on the interaction between response rates and conversion rates for tourist conversion surveys, Woodside and Dubelaar (2003) hypothesize and confirm that identifying one destination-brand as the sponsor of the survey decreases the response rate while inaccurately increasing the conversion rate. Such findings show that the methods to increase response rates may jeopardize the validity of the survey results. Thus, Woodside and Dubelaar (2003) advocate experimental studies on the manipulation of contextual cues in order to test the validity of conversion studies.

This study responds to Woodside and Dubelaar’s (2003) recommendation by testing the effects of contextual cues on response and conversion rates of online studies. The study here applies an elaborate experimental design. Specifically, the study manipulates three contextual cues—researcher identity, sponsor identity, and incentive type—and tests their effects on response and conversion rates of online conversion studies.

Theoretical Frameworks of Survey Responses

Researchers propose a few major theoretical frameworks to explain nonresponses and their effects on survey results. Groves, Cialdini, and Couper (1992) suggest that societal factors, interviewer attributes, and respondent–interviewer interaction directly affect survey response rates. The psychological factors of the respondents may in turn influence those factors, including compliance with requests, helping tendencies, and opinion change. Groves, Singer, and Corning (2000) propose the leverage-saliency theory to explain people’s decision in responding to a survey. Many attributes influence their decisions to respond, including incentive, the survey senders, the sponsors, and the topic. Survey receivers implicitly assign different direction (positive or negative) and weights to each attribute. The summed value of the multiplication of the weight and direction of each attribute affect whether or not a survey receiver responds. Groves et al. (2006) apply response propensity to define the likelihood and probabilities of being a respondent. Nonresponse bias occurs when a nonzero covariance exists between a survey variable and the respondent propensity. Nonresponse bias can vary across different statistics obtained from a survey. They use randomized experiments to test if respondents who are interested in the survey topic are likely to respond. The results partially confirm a positive bias in that many surveys may result in exaggerated population estimates and prevalence of interests or activities (Groves et al. 2006).

Stalans (2012, p. 75) explains the effects of context on survey responses and response rates using the psychological concept of “frame.” A frame includes “the essence of the story, the central guiding theme, or a salient strategy that suggests how a decision should be made.” Survey introductions, certain phrases in a survey question, or survey sender identity, could become frames through which the survey respondents interpret the meaning and intent of specific questions and the whole survey. Respondents may assign greater importance to the highlighted message in the question and form their opinion about the issue accordingly. These frames, including survey sponsorship or survey introduction, are part of contextual effects that create meanings and may encourage or deter certain participants to respond. Thus, the framing effect can create the covariance between response propensity and certain variables in the survey results. Groves et al. (2006) experimentally test and partially validate the framing effect of topic of interests and how framing influences response rates and affects survey results.

However, no experiment to date examines more subtle contextual cues such as sponsorship, survey sender identity, or incentive’s effects on the interaction between response rates and survey bias (Woodside and Dubelaar 2003). Studies frequently incorporate these contextual cues with the hope of boosting response rates. First, studies show that survey senders in high power, either the researcher or the sponsoring body, generate more responses than low-power and less authoritative requestors (Joinson and Reips 2007; Porter and Whitcomb 2003; Wright and Schwager 2008). When a specific business or organization sponsors advertising conversion studies conducted on their existing customer database, they would like to reveal their identity in the hope of increasing response rates. Second, educational institutions as survey senders can also solicit more responses (Jobber and O’Reilly 1998; Sheehan and McMillan 1999). Those businesses or organizations often hire educational institutions to conduct surveys since the latter are more authoritative and usually generate a higher response rate. Third, providing free product samples or sweepstakes from the business as survey-response incentives is convenient and frequently used in some studies (e.g., Wilcox and Woodside 2012). However, revealing these contextual cues (the research body, sponsorship, or incentive) about the survey may simultaneously affect the conversion rates (Woodside and Dubelaar 2003).

Surveyors have a natural but unjustified tendency to maximize response rates as much as possible (Groves 2006). However, the increase in response rates may be unjustifiable if the increase jeopardizes the validity of survey results (Groves 2006). Inconsistency in past studies may be attributable to specific circumstances or scenarios under which each study was conducted (Groves, Singer, and Corning 2000). In addition, few studies use an experimental design to target the same population with the same method and to test the intertwining effects of online response rates and biased results due to revealed contextual cues. Therefore, in particular, this study aims to test the effects of contextual cues on response rates and conversion rates of online conversion studies.

Method

To investigate the contextual cues’ influence on response rates and conversion rates, the study includes a between-subjects and fixed-effect factorial design with three factors (Campbell, Stanley, and Gage 1969). The experimental design applies to an online conversion study for the website visitors who opted to receive email alerts from City A’s Convention and Visitors Bureau (CVB) website. City A is a tourist city in the southeast United States. The survey’s intent is to gauge the conversion rates and other demographic and behavioral characteristics of the information inquirers. The demographic variables include age, gender, home location, household size, education, employment status, and income; behavioral variables include travel history to each of the eight cities in Southeast United States, and information regarding their most recent trip, including travel party, transportation, length of stay, accommodation type, trip purpose, decision criteria, information source use, satisfaction, and re-visit intention.

The experimental study manipulates the following three contextual cues. Two different survey senders include researcher A with a foreign name from a regional university in City A, and researcher B with an American-Christian name from a national university in a distant city from City A. Two different sponsors include City A’s CVB, and an international academy of tourism and hospitality. Two incentives for lottery drawing include one unnamed hotel stay in City A, and an unnamed hotel stay from one of the eight Southeast cities in the United States. The eight cities include City A and its seven competing cities: Asheville, North Carolina; Atlanta, Georgia; Charleston, South Carolina; Hilton Head, South Carolina; Myrtle Beach, South Carolina; Savannah, Georgia; St. Augustine, Florida; and Wilmington, North Carolina. All first conditions of the three factors include the name of the specified city, City A. This results in a 2 × 2 × 2 full factorial design with eight different cells (Tables 1 and 3).

OPEN IN VIEWER

Table 1.

Three Factors: Researcher Identity, Sponsor Identity, and Incentive Type.

Factors	First Condition	Abbreviations	Second Condition	Abbreviations
Researcher	Researcher A in a City A university	Researcher A	Researcher B in a national university	Researcher B
Sponsor	City A convention and visitors bureau	CVB sponsor	An international academy of tourism and hospitality	Academy sponsor
Incentive	A hotel in City A	City A incentive	A hotel in one of eight cities in Southeast United States	SE incentive

OPEN IN VIEWER

Table 2.

Tests of Conversion Rates between the First/Second Wave versus the Third Wave.

	First and Second Wave	Third Wave	Chi-Square	Significance
Total returned responses	1,257	71	–	–
Conversion rate to City A for ever visited	76.1%	69.0%	1.487	0.2226
Conversion rate to City A for past six months	37.2%	31.0%	0.849	0.3568
Conversion rate to City A for the most current trip	36.8%	19.7%	7.926	0.0049

OPEN IN VIEWER

Table 3.

Response Rates by Researcher Identity, Sponsor Identity, and Incentive Type in Eight Cells.

Condition	Details	Total Sample	Total Respondents	Response Rate (%)
Cell 1	Researcher A, CVB sponsor, City A incentive	2177	158	7.3
Cell 2	Researcher A, CVB sponsor, SE incentive	2177	145	6.7
Cell 3	Researcher A, academy sponsor, City A incentive	2177	128	5.9
Cell 4	Researcher A, academy sponsor, SE incentive	2177	137	6.3
Cell 5	Researcher B, CVB sponsor, City A incentive	2177	197	9.0
Cell 6	Researcher B, CVB sponsor, SE incentive	2177	172	7.9
Cell 7	Researcher B, academy sponsor, City A incentive	2176	161	7.4
Cell 8	Researcher B, academy sponsor, SE incentive	2177	159	7.3

Adopting the recommendations by Woodside and Dubelaar (2003), the researchers did not remind the respondents that they had joined the email alert from the CVB website. Instead, the respondents were asked to report on whether or not they have visited any of the eight cities during three time frames: the first question asked the respondent to select the cities they had ever visited among the eight cities listed; the second question requested the respondents to identify the cities they visited in the past six months; the third question asked about the city of their most recent trip, regardless whether the city is one of the eight. The three questions can be calculated as different versions of conversion rates to City A.

A total of 17,415, or one in five, active email addresses were randomly selected from all the visitors who opted to receive weekly email alerts from City A’s CVB website. The researchers designed eight different survey forms on SurveyMonkey.com and eight different corresponding email solicitation messages. The 17,415 email addresses were equally and randomly assigned to one of the eight cells. Seven of eight cells have 2,177 solicitation emails sent out, except one with 2,176. Most of the content of the solicitation emails was the same other than the statement of the three variable factors: researcher, sponsor, and incentive (see Figure 1 for the manipulation).

OPEN IN VIEWER

Figure 1.

Solicitation email messages with three factors: researcher identity, sponsor identity, and incentive type.

An opening paragraph in the actual survey reiterated the content in the solicitation emails with these three different factors. A custom script sent out the email messages and corresponding links to the email respondents (Figure 1). The first wave of email solicitations were distributed on September 7, 2010, with a customized computer script. In order to increase the response rates, two days after the first wave of solicitation emails, the script sent one follow-up wave of solicitations to those email addresses who had not responded.

The first and second wave of responses generated a total response rate of 7.2%. Thus, one may question the validity of the survey results since the response rate is low, and the respondents may represent a different sub-population. The “continuum-of-resistance” model argues that the respondents who require multiple solicitations before responding more closely resemble those non-respondents, since those people would be non-respondents if the final wave of solicitation had not been sent out (Filion 1976). Therefore, the researchers conducted the third wave of email solicitation following two months after the first and second waves. A total number of 1,600 email addresses were randomly selected from the non-respondents for the first two waves, with 200 emails in each cell. The results can be compared to those collected from the first two waves in order to assess the validity of the conversion rates of the first two.

Results

One month after the first and second waves of solicitations, less than 100 emails were returned as “undeliverable” or bounced back as an “out of office” message. There is no reason to suspect that the bounced-back emails are nonrandom among the eight cells. In addition, the total number of undelivered messages was relatively insignificant compared to the total sample. Thus, the researchers calculated the gross response rate, which does not consider nondeliverable surveys among eight cells when calculating response rates (Deutskens et al. 2004). Since the first and second waves of email solicitations were sent within a short period of time of each other, they were analyzed as one data set.

One month after the third wave of solicitation, a total of 71 surveys were returned for all eight cells. The response rates and conversion rates were aggregated and compared to those collected from the first and second waves in order to assess the validity of the conversion rates.

Response Rates

The data from the first two waves were analyzed to test the conditions that may influence the response rates in the eight cells (Table 3). The response rates varied from 5.9% to 9.0%. Multiple chi-square tests were used to assess the significance of the three factors on response rates: researcher, sponsor, and incentive type (Table 4). The results show that researcher and sponsor both significantly affected the response rate (p < 0.01), but not the incentive type. Thus, different researcher identity and sponsor identity would significantly influence potential respondents’ decision to reply.

OPEN IN VIEWER

Table 4.

Tests of the Impact of Researcher Identity, Sponsor Identity, and Incentive Type on Response Rates.

Type	Chi-Square Value	Significance
Sponsor	9.965	0.002**
Researcher	7.602	0.006**
Incentive	0.653	0.419

**

Significant at 0.01 level.

Table 3 shows that researcher A in low power generated a lower response rate compared to researcher B in high power; thus, hypothesis 1a receives support. However, contrary to hypothesis 2a, a regional and familiar brand sponsor (the respondents all requested information from the CVB website) actually increased the response rate over a national brand sponsor. Thus, the findings do not support hypothesis 2a and suggest a direction opposite to the hypothesis. In this study, City A’s CVB as the sponsor is a familiar organization to respondents, since all respondents had requested information from its website, whereas the national academy sponsor may be unfamiliar and thus implies less credibility. Nowadays, a consumer receives increasing amount of spamming. Thus, the judgment of the credibility of online surveys may be of paramount importance, based on the power status of the survey sender and the familiarity of the sponsoring body. In addition, a regional incentive on a specific city did not increase response rates significantly more than a general incentive in a wider geographical area. Thus, the findings fail to support hypothesis 3a.

In order to understand the interaction of the factors, a logistic regression was further used to regress whether or not one responds to the survey on the three contextual cues. The results find no significant interaction effects between these three factors. Researcher B garnered 21.3% more responses than Researcher A. Furthermore, City A’s CVB as the sponsor generated a 14.9% increase in response rate compared to the national academy sponsor. Combining the two factors of researcher and sponsor, the response rate could increase up to 39.2% (Figure 2).

OPEN IN VIEWER

Figure 2.

The impact of researcher identity, sponsor identity, and incentive type on response rates.

In general, the results show that the response rates of this particular online survey were extremely low, an average of 7.2 percent, confirming the concerns of low response rates of online surveys. The contextual cues given away by the researcher’s identity and sponsoring body influenced the response rates, and the combination of researcher identity and sponsor identity could increase the response rate up to 39%. In conclusion, hypothesis 1a receives support: the researcher in a high-power position generates more responses than one in low-power. Hypotheses 2a and 3a fail to receive support: a low-power but familiar sponsor generates more responses than a high-power but unfamiliar sponsor (hypothesis 2a in the opposite direction). The type of incentive does not affect response rates significantly (hypothesis 3a).

Conversion Rates

The three types of conversion rates to City A from the first two waves of responses by three factors were tabulated (Table 5). No clear patterns were observed and a logistic regression on the prediction of being one of converted visitors by the three factors did not yield significant results (Table 6). Thus, the conversion rates, in all three different measurements, were unaffected by the three contextual cues. Therefore, contrary to proposed hypotheses, researcher identity, sponsor identity, or different incentives, did not influence conversion rates significantly. Thus, the findings do not support hypotheses 1b, 2b, and 3b.

OPEN IN VIEWER

Table 5.

Three Types of Conversion Rates Under Three Factors: Researcher Identity, Sponsor Identity, and Incentive Type.

Researcher			Incentive (%)
Have ever visited City A			City A	Southeast	Average (%)
A	Sponsor	Academy	75.8	78.1	74.1
		CVB	74.1	80.7	76.4
	Average		74.8	79.4	77.1
B	Sponsor	Academy	72.0	76.1	77.0
		CVB	75.6	77.3	77.2
	Average		74.0	76.7	75.3
Visited City A within six months
A	Sponsor	Academy	41.4	40.9	35.0
		CVB	34.8	40.0	36.0
	Average		37.8	40.4	39.1
B	Sponsor	Academy	38.5	31.4	41.1
		CVB	36.0	36.0	37.3
	Average		37.2	33.8	35.6
Visited City A in the most recent trip
A	Sponsor	Academy	46.1	40.1	33.4
		CVB	36.1	37.9	35.2
	Average		40.6	39.0	39.8
B	Sponsor	Academy	34.8	32.1	43.0
		CVB	40.1	29.7	37.0
	Average		37.7	30.8	34.4

OPEN IN VIEWER

Table 6.

Testing the Impact of Researcher Identity, Sponsor Identity, and Incentive Type on Conversion Rates with Logistic Regression.

	Have Ever Visited City A		Visited City A in Previous Six Months		Visited City A in the Most Recent Trip
	Value	Significance	Value	Significance	Value	Significance
Incentive	2.460	0.227	0.355	0.552	0.056	0.813
Sponsor	1.049	0.306	0.062	0.803	1.566	0.211
Researcher	0.013	0.910	3.129	0.077	1.554	0.213

Contrary to the hypotheses, the contextual cues explicitly or implicitly associated with the product did not attract more converted consumers to reply. The researchers suspect that most recipients did not bother to respond to the surveys because of reasons other than topic interests, such as busy schedule and increasing amount of emails and surveys. The added familiarity to the destination under certain contextual cues did not create enough weight to change the direction of the formula of responding, according to the leverage-saliency theory (Groves, Singer, and Corning 2000).

Discussion

This study reveals low response rates of online conversion studies. Some hypotheses regarding response rates were rejected while others receive support. A high-power researcher and a familiar sponsor generated higher response rates, while a familiar incentive did not. The three contextual cues did not affect three types of conversion rates.

The study shows that a researcher in a high-power position could generate more responses than one in lower power position. However in this study, the researcher affiliated with the regional university is from an ethnic minority and has a typical non-American name, while the researcher from the national university is a Caucasian with a typical American-Christian name. Prior research reports that the name of the survey sender influences the response rate, as senders with American-Christian names generate a higher response rate than those with foreign non-Christian names (Chawla and Nataraajan 1994).

Therefore, as important components in the researcher identity, both the researchers’ names and their university affiliation, may affect the response rate. Whether or not only one factor leads to the different response rate or the combination of the two influences the response rates is impossible to conclude. However, both researchers’ names were associated with the title of “Ph.D.” As Chawla and Nataraajan (1994) discuss, with added titles, the differences in bias due to names may disappear. Thus, the researchers estimate that the national university affiliation of the second researcher may have contributed more to the higher response rates.

The two sponsors of this study include one local CVB and one national academic organization. The respondents in this study are all information seekers who opted to receive email alerts from this particular local CVB; therefore, they are more familiar with this sponsor than the national academic organization. The embedded three features of the sponsorship in this study—a familiar versus unfamiliar organization, a local versus national one, and an industrial versus academic one—may individually or collectively influence the response rates. Past studies show that academic institutions could garner more responses than industrial ones (Sheehan and McMillan 1999); also the organization with a higher power status (national vs. regional) would result in a higher response rate. This study shows the opposite: the local CVB sponsor generated a higher response rate than the national academic organization. This finding confirms the value of familiar sponsors, since the CVB sponsor even overcame the disadvantages of being an industrial and regional organization.

Implications

This study touches on two priorities in tourism research: the appropriate measurement of destination marketing strategies and the methodological validation of marketing programs (Williams, Stewart, and Larsen 2012). Since today’s online surveys suffer severely from low response rates, marketing researchers need to carefully design survey content and delivery methods to maximize survey response rates; however, this effort should not at the same time compromise the quality of survey results and create more survey biases.

The results of this study suggest or validate the following guidelines. First, researchers need to make every effort to construct an authoritative, trustworthy, and consistent solicitation message. One should adopt a researcher higher in power, higher in organizational position or educational achievement, or serving in a national-level educational institution, as the main contact person. Whenever possible, it is the best to list the formal title and degree of the researcher to validate the authority.

Second, if applicable, the research team should use a researcher who has a name in similar cultural background as the respondents’ to be the main contact person. For example, use a Chinese researcher’s name as the contact person when surveying a Chinese immigrant group. Third, researchers should use and disclose a sponsor’s information that the respondents are familiar with.

Despite all suggestions to increase response rates, it seems unlikely that these recommendations for online-marketing effectiveness studies will increase response rates dramatically. Even though this study did not reveal significant bias in conversion rates, and low response rate might not necessarily lead to low quality results (Groves 2006). Not every conversion study can afford the time and effort to conduct extensive analysis on survey bias on the specific survey. Thus, whether or not a conversion study is accurate may remain unknown.

Consider applying the following broad guidelines in conducting conversion studies. First, offer free and useful material as long as the prospect is willing to provide her or his telephone number for future two-minute interviews; when conducting the random interviews, make a few attempts to reach the respondents by telephone. Even though the response number is likely to be low because of the high cost, the obtained conversion rates are likely to be more meaningful and accurate.

Second, a mixed-mode survey method serves to maximize response rates. Millar and Dillman (2011) report that some specific techniques are useful for improving online response rates. For example, researchers can consider email solicitation first and a mail follow-up option in the final contact, or combine both postal and email contacts, and deliver a token cash incentive in advance. Further studies are necessary to explore and validate different means of online and mixed-mode data collection to combine multiple response-inducing strategies.

Third, in reality, many other factors rather than its scientific values may influence a conversion study. One of the authors of this study conducted a conversion study where consumption data were collected without identifying any brand. The sponsor was dissatisfied with the findings, which included a 25 percent increase in the survey response rate and a 25 percent decrease in the conversion rates in comparison to similar studies by competing brands (Arch Woodside, Personal communication, June 2011). In this instance, the sponsoring brand’s manager received these findings with displeasure. The brand manager preferred being able to report conversion rates equal to the conversion rates in the reports made by competing brands.

One may argue that if every destination conducts conversion studies by identifying the sponsors, the inaccurate conversion rates are actually comparable on the same methodology across all competing destinations. However, the identified sponsor may affect the conversion rates in different ways and magnitude, and without rigorous testing, one may never know all these effects to justify that approach. As a result, researchers should adopt a conservative approach with suggested recommendations here. The scientific value of a rigorous approach, rather than a politically motivated one, will be more beneficial to the fields of tourism research and tourism marketing in the long run.