Acceptance and coverage of fast invitation methods to in-the-moment surveys

Introduction

Despite being a major tool of empirical investigation (Saris & Gallhofer, 2014), surveys are affected by errors (Weisberg, 2005) that can compromise the results of the research (Saris & Revilla, 2016). Some of these errors are produced by the respondents’ inability to properly recall past events (Tourangeau, 1999). Recall errors increase with time, rapidly at first, and then more gradually (Dallenbach, 1913). Therefore, surveying participants closer to an event of interest should reduce recall errors (for further information about the advantages of this method, see Ochoa et al., 2016).

Despite the little scientific research on this method, in-the-moment surveys are currently used. Some examples are the surveys of voters leaving polling stations to predict an election outcome (Frankovic, 2012), or popup web surveys of people buying flight tickets online to know their satisfaction with the purchase process. In-the-moment surveys have also proved their value in audience measurement (Lamas, 2005) and psychological research (van Berkel et al., 2017). However, current examples, both offline and online, correspond to very specific situations where the detection of individuals experiencing the event of interest and the feasibility of surveying them are particularly convenient.

Online panels (individuals participating in web surveys, usually in exchange of rewards) offer an opportunity to extend the use of in-the-moment surveys to new research problems. Some of these panels (e.g., Gapfish, Netquest, Respondi, and Yougov) also ask panellists to install tracking software (a meter) in their browsing devices to passively collect information about their online behavior (e.g., visited URLs). Such metered panels (see Revilla, Couper et al., 2021) can be used to detect online events of interest and trigger a web survey invitation right in the moment: for instance, to investigate the opinions of individuals who have just read an online newspaper or purchased a product online.

The benefits of in-the-moment surveys can only materialize if panellists accept to participate. Ochoa and Revilla (2021) found notably high levels of willingness to participate among members of a metered panel in Spain, ranging from 69% to 95% (93% on average) depending on the survey features. Nevertheless, the willingness to participate may differ substantially from the actual participation due to several reasons. One critical problem is that participants may not see the invitation to participate in time.

This might lead to a high non-participation even within a sample of panellists willing to participate. Thus, the method used to invite participants to answer a survey is crucial. We define an invitation method as a combination of (1) a messaging system used to send the invitation (e.g., email or SMS) and (2) a participant’s device used to receive it (e.g., smartphone or PC).

When online panels first appeared in the mid-90s (Callegaro et al., 2014), the email-PC combination was almost the only invitation method available. With the arrival of the mobile Internet, panellists started to access web surveys by checking their emails from their smartphones, leading to a new combination (email-smartphone). Since then, new devices such as tablets have appeared, and several new messaging systems have been developed. Some of them have been specifically designed to work on smartphones and tablets, such as panel apps (that can show instant push notifications) and WhatsApp (an instant messaging system widely used worldwide ¹ ). These new messaging systems, in combination with the existing devices, can potentially be used as invitation methods.

To reduce the risk of panellists not seeing the survey invitation in time when implementing in-the-moment surveys, online panels should consider using different invitation methods. Because these invitation methods might not be the same as for conventional surveys, they might require additional acceptance from participants. Accepting a new method may, in turn, require installing software on the participants’ devices (e.g., an app on their smartphones or a plug-in on their PCs) or at least granting additional permissions to existing software (e.g., allowing an existing app to show push notifications). To the best of our knowledge, there is no previous research about the acceptance of different invitation methods. In addition, within the accepted methods, those most likely to be seen first by respondents should be preferred to increase the likelihood of the invitation being seen in time. However, little is known about which methods are the “fastest.”

To start filling these gaps, in this study, we explore for members of an opt-in metered panel in Spain, the acceptance and coverage of eight invitation methods. Acceptance is defined as the number of panellists accepting a method over those who met the requirements to be offered such method (see Section 2)

Acceptance = Panellists accepting a method Panellists offered a method

Coverage = Panellists accepting a method Total number of panellists ≈ Acceptance × Device use

Finally, we explore what methods panellists think they would see first among those accepted (“fastest” invitation methods).

Method and data

Results

Acceptance and coverage of different invitation methods

On average, respondents accept 4.2 invitation methods (69.1% of the methods offered to them). Moreover, 77.6% of them accept three or more invitation methods.

Table 1 shows the number of respondents stating that they would accept each method (column “No. Accept”) and the sample size per category (column “N”; this changed in accordance with their use of devices and, in the case of popup-PC, stated acceptance of being invited when detecting an event of interest through their PC). The table also presents the acceptance and coverage, as defined in Section 1.

OPEN IN VIEWER

Table 1.

Acceptance and coverage of different invitation methods.

Invitation method	No. Accept	N	Acceptance (%)	Coverage (%)
App-smartphone	685	786	87.2	86.3
Email-smartphone	645	786	82.1	81.2
WhatsApp-smartphone	492	786	62.6	62.0
SMS	457	792	57.7	57.6
Popup-PC	400	566	70.7	50.4
App-tablet	267	367	72.8	33.6
Email-tablet	254	367	69.2	32.0
WhatsApp-tablet	162	367	44.1	20.4

App-smartphone is the most accepted method, followed closely by email-smartphone, while WhatsApp-tablet is the least accepted one. Regarding coverage, as it strongly depends on the level of use of the device, the three smartphone methods are the ones with the greatest levels, followed by SMS and Popup-PC. Although SMS is poorly accepted, it works in any mobile phone, which produces a coverage similar to its acceptance level.

Next, in order to better see the role played by the devices and messaging systems, Table 2 shows the average acceptance and coverage of the methods grouped first by device and second by messaging system.

OPEN IN VIEWER

Table 2.

Acceptance and coverage of invitation methods by device and messaging system.

	Average acceptance (%)	Average coverage (%)
By device
Smartphone	77.3	76.5
PC	70.7	50.4
Tablet	62.0	28.7
By messaging system
App	82.6	59.9
Email	78.0	56.6
WhatsApp	56.7	41.2
Popup (PC)	70.7	50.4
SMS (Mobile)	57.7	57.6

Tablet is the least accepted device to receive invitations, whereas smartphone is the most accepted one (1.3 times more). Such difference, in combination with the level of use of each device, translates into an even larger gap in coverage (2.7 times). Furthermore, WhatsApp is the least accepted messaging system and the app the most accepted one (1.5 times more), whereas the difference in coverage remains proportional as both messaging systems were offered in combination with smartphone and tablet. Finally, as SMS was not linked to a specific device and popup was always linked to PC, their acceptance and coverage remain the same as shown in Table 1.

Combined coverage of different invitation methods

Next, we study what set of methods would be accepted by most panellists. Table 3 shows which methods maximize the combined coverage for any given number of methods.

OPEN IN VIEWER

Table 3.

Combined coverage.

Number of invitation methods	Invitation method	Combined coverage (%)	Incremental coverage (%)
1	App-smartphone	86.3	86.3
2	+ Email-smartphone	91.6	5.3
3	+ Popup-PC	95.1	3.5
4	+ SMS	96.3	1.3
5	+ WhatsApp-smartphone	96.7	0.4
6	+ Email-tablet	97.0	0.3
7	+ App-tablet	97.0	0.0
8	+ WhatsApp-tablet	97.0	0.0

If only one method could be offered, it should be app-smartphone. If a second method could be offered, email-smartphone should be chosen as it is the one adding most coverage (i.e., 5.3% of panellists accept this method but do not accept app-smartphone). Popup-PC should be the third method offered (+3.5% coverage) and SMS the fourth (+1.3%). The remaining methods barely add coverage.

Fastest invitation method

In order to assess, within the accepted methods, which one is most likely to be seen in time by participants, Table 4 shows the number of respondents stating which invitation method they would see first ⁵ (column “No. First”) and its proportion with respect to the number of people accepting each method (column “No. First/No. Accept”). The column “Share” shows the percentage of the fastest invitation method over the total participants. Lastly, the column “Combined share” presents the accumulated percentage to give insight into how adding invitation methods (starting from the fastest one) would increase the percentage of panellists who are offered the method they consider the fastest.

OPEN IN VIEWER

Table 4.

Fastest invitation method.

Method	No. First	No. First/No. Accept (%)	Share (%)	Combined share (%)
App-smartphone	243	35.5	33.7	33.7
WhatsApp-smartphone	204	41.5	28.3	62.0
Email-smartphone	123	19.1	17.1	79.1
Popup-PC	74	18.5	10.3	89.4
SMS	47	10.3	6.5	95.9
App-tablet	12	4.5	1.7	97.6
Email-tablet	11	4.3	1.5	99.1
WhatsApp-tablet	7	4.3	1.0	100

The three invitation methods based on smartphone (the most used device) are the ones that most participants stated they would see first: App-smartphone is the fastest followed by WhatsApp-smartphone, despite being the third least accepted method. This is because WhatsApp is accepted by relatively few panellists, but most of them consider this method the fastest (highest No. First/No. Accept ratio).

Regarding the combined share, if the three smartphone methods were offered, 79.1% of panellists would be using a method that they consider the fastest one. Adding Popup-PC and SMS raises this figure up to 95.9%.

Discussion

In-the-moment surveys can be used to research a variety of topics where memory issues are relevant. However, such surveys suppress one key advantage of web surveys: the possibility for participants of choosing their own schedule for participating (Albaum & Smith, 2012). Consequently, panellists willing to participate may not ultimately do so for several reasons such as lack of time or motivation. But these factors become of secondary importance if participants do not see the survey invitation in time. To that end, fast but acceptable invitation methods should be used.

The current results provide evidence on what invitation methods, among those considered suitable for in-the-moment surveys, are most accepted by panellists of a metered panel in Spain. Receiving invitations in a smartphone using a panel app is the most accepted one. This method is also the one that most panellists consider they would see first. Thus, if only one invitation method is to be used, we recommend using App-smartphone.

However, according to the results of this research, panellists would accept several methods. Thus, when the available time to participate is so critical as in in-the-moment surveys, we recommend that panel companies use multiple invitation methods simultaneously. To select the set of methods offered to panellists, two main aspects should be considered: coverage and speed. Coverage ensures that the maximum number of panellists accept to use one of the offered methods. Speed reduces the risk that panellists who are willing to participate do not see the invitation in time.

Showing invitations by means of an app installed by participants in their smartphones contributes the most to both coverage and speed. That suggests that panels considering conducting in-the-moment surveys should have an app as a starting point, although persuading panellists to install it may be challenging (Revilla, Paura, & Ochoa, 2021). As a second step, encouraging panellists to install an email app on their smartphones (in case they do not have it yet) and activate the push notifications when a new email arrives would significantly contribute also to coverage and speed, without any technological development (panels generally use email as invitation method).

At this point, different directions should be taken to further improve coverage and speed. Regarding coverage, combining smartphone-based methods with one or two alternatives based on PC and/or SMS could be a successful strategy to maximize the coverage of potential participants. Regarding speed, using messaging systems such as WhatsApp should be seriously considered. Although only less than 6 out of 10 participants would accept this method, it is considered as the fastest method by 41% of them (28% of all panellists). However, WhatsApp is currently limiting the maximum number of recipients of a message, even for companies and institutions, which reduces its feasibility as an invitation method.

In any case, offering at least one invitation method per device increases the likelihood of inviting a panellist using the same device where the event of interest is detected, which may contribute to reduce the time needed to see the invitation.

All the above recommendations have in common the need to adapt to participants. Getting people to participate in surveys has been increasingly difficult over the last years (Leeper, 2019). In-the-moment surveys ask panellists to participate under time pressure, which is expected to reduce participation rates. Researchers interested in investigating in the moment should offer new methods to invite panellists, allowing them to choose the more convenient ones. Despite the evidence that combining different invitation methods should be effective in reaching participants in time, overwhelming panellists with repeated messages through different channels could end up with an increased fatigue, leading to panel attrition in the long run. Faced with this dilemma, panel companies need to redesign their incentivization policies to adapt to a reality that is far more complex than rewarding conventional web surveys. Further research is needed to determine the best ways to do so. Finally, the findings presented in this research note are based on a specific sample, country, and kind of in-the-moment surveys. Further research is needed to test the robustness, especially for surveys triggered by different data (e.g., geolocation data), different kinds of samples (e.g., probabilistic panels), and countries. Besides, the reason why some invitation methods are better accepted than others deserves further research. Privacy issues may play a key role in participants’ preferences. Such understanding should allow to design new invitation methods in the future or, at least, communicate the existing ones more effectively.