Abstract
When asking survey participants about past events, respondents might not properly recall the requested information. Surveying participants right when an event of interest occurs should reduce these recall errors.
Such “in-the-moment surveys” are used nowadays but only in very specific occasions. Online panels that ask their members to share their online behaviors (metered panels) offer a new opportunity to use in-the-moment surveys whenever an online event of interest is detected.
Previous research shows that the willingness to participate in in-the-moment surveys is notably high in metered panels, but even panellists willing to participate may fail to do so if they do not see the invitation in time. Very little is known about how participants perceive the different invitation methods available.
A survey of members of a metered panel in Spain reveals that invitation methods deployed on smartphones get higher levels of acceptance and coverage, and are perceived as fastest. Moreover, offering several invitation methods on different devices would maximize the opportunities to participate in time, making them more feasible.
Introduction
Despite being a major tool of empirical investigation (Saris & Gallhofer, 2014), surveys are affected by errors (Weisberg, 2005) that can compromise the results of the research (Saris & Revilla, 2016). Some of these errors are produced by the respondents’ inability to properly recall past events (Tourangeau, 1999). Recall errors increase with time, rapidly at first, and then more gradually (Dallenbach, 1913). Therefore, surveying participants closer to an event of interest should reduce recall errors (for further information about the advantages of this method, see Ochoa et al., 2016).
Despite the little scientific research on this method, in-the-moment surveys are currently used. Some examples are the surveys of voters leaving polling stations to predict an election outcome (Frankovic, 2012), or popup web surveys of people buying flight tickets online to know their satisfaction with the purchase process. In-the-moment surveys have also proved their value in audience measurement (Lamas, 2005) and psychological research (van Berkel et al., 2017). However, current examples, both offline and online, correspond to very specific situations where the detection of individuals experiencing the event of interest and the feasibility of surveying them are particularly convenient.
Online panels (individuals participating in web surveys, usually in exchange of rewards) offer an opportunity to extend the use of in-the-moment surveys to new research problems. Some of these panels (e.g., Gapfish, Netquest, Respondi, and Yougov) also ask panellists to install tracking software (a meter) in their browsing devices to passively collect information about their online behavior (e.g., visited URLs). Such metered panels (see Revilla, Couper et al., 2021) can be used to detect online events of interest and trigger a web survey invitation right in the moment: for instance, to investigate the opinions of individuals who have just read an online newspaper or purchased a product online.
The benefits of in-the-moment surveys can only materialize if panellists accept to participate. Ochoa and Revilla (2021) found notably high levels of willingness to participate among members of a metered panel in Spain, ranging from 69% to 95% (93% on average) depending on the survey features. Nevertheless, the willingness to participate may differ substantially from the actual participation due to several reasons. One critical problem is that participants may not see the invitation to participate in time.
This might lead to a high non-participation even within a sample of panellists willing to participate. Thus, the method used to invite participants to answer a survey is crucial. We define an invitation method as a combination of (1) a messaging system used to send the invitation (e.g., email or SMS) and (2) a participant’s device used to receive it (e.g., smartphone or PC).
When online panels first appeared in the mid-90s (Callegaro et al., 2014), the email-PC combination was almost the only invitation method available. With the arrival of the mobile Internet, panellists started to access web surveys by checking their emails from their smartphones, leading to a new combination (email-smartphone). Since then, new devices such as tablets have appeared, and several new messaging systems have been developed. Some of them have been specifically designed to work on smartphones and tablets, such as panel apps (that can show instant push notifications) and WhatsApp (an instant messaging system widely used worldwide 1 ). These new messaging systems, in combination with the existing devices, can potentially be used as invitation methods.
To reduce the risk of panellists not seeing the survey invitation in time when implementing in-the-moment surveys, online panels should consider using different invitation methods. Because these invitation methods might not be the same as for conventional surveys, they might require additional acceptance from participants. Accepting a new method may, in turn, require installing software on the participants’ devices (e.g., an app on their smartphones or a plug-in on their PCs) or at least granting additional permissions to existing software (e.g., allowing an existing app to show push notifications). To the best of our knowledge, there is no previous research about the acceptance of different invitation methods. In addition, within the accepted methods, those most likely to be seen first by respondents should be preferred to increase the likelihood of the invitation being seen in time. However, little is known about which methods are the “fastest.”
To start filling these gaps, in this study, we explore for members of an opt-in metered panel in Spain, the acceptance and coverage of eight invitation methods. Acceptance is defined as the number of panellists accepting a method over those who met the requirements to be offered such method (see Section 2)
Finally, we explore what methods panellists think they would see first among those accepted (“fastest” invitation methods).
Method and data
Data
The data were collected in Spain in May/June 2021 through a web survey conducted on the Netquest metered panel (http://www.netquest.com). Members of this panel are rewarded for both participating in surveys and sharing metered data, after installing the meter in at least one browsing device. Metered panellists are overall more experienced than non-metered Netquest panellists. The panellists analyzed in this study have been in the panel for 7.6 years on average, completing approximately one survey per week.
Quotas for gender, age, and education were defined to reproduce the proportions of the online population in Spain according to the Spanish National Statistics Institute. 2 1,900 panellists were invited to the survey, and 1,701 started it (89.5%). After discarding participants for exceeding the quotas, not giving their explicit consent to participate, not passing basic anti-fraud checks, and not being willing to participate in in-the-moment surveys, we analyzed 794 panellists. Their average age was 45 years. 50% were women. 25% were mid-educated, and 35% high educated. 68% installed the meter in a desktop device, 86% in a smartphone, and 30% in a tablet.
Questions of interest
The questionnaire
3
mainly focused on assessing the willingness to participate in in-the-moment surveys triggered by metered data. The questions of interest were: • • PC (Yes/No) • Smartphone (Yes/No) • Tablet (Yes/No) • • … PC? (Yes/No) • … smartphone? (Yes/No) • … tablet? (Yes/No) • • (Email-smartphone) … an email received on your smartphone, activating the instant notification? (Yes/No) • (Email-tablet)… an email received on your tablet, activating the instant notification? (Yes/No) • (App-smartphone) … an instant message sent by a Nicequest app installed on your smartphone? (Yes/No) • (App-tablet)… an instant message sent by a Nicequest app installed on your tablet? (Yes/No) • (WhatsApp-smartphone)… a message sent via WhatsApp to your smartphone? (Yes/No) • (WhatsApp-tablet) … a message sent via WhatsApp to your tablet? (Yes/No) • (SMS)… an SMS? (Yes/No) • (Popup-PC) … a popup window shown on your PC while browsing? (Yes/No) • • … an email received on your smartphone, activating the instant notification? • … an email received on your tablet, activating the instant notification? • … an instant message sent by a Nicequest app installed on your smartphone? • … an instant message sent by a Nicequest app installed on your tablet? • … a message sent via WhatsApp to your smartphone? • … a message sent via WhatsApp to your tablet? • … an SMS? • … a popup window shown on your PC while browsing? • None
The question “Device” was used to tailor the answer options in the questions “TriggeringDevice” (only showing the devices regularly used) and “Method” (only showing the invitation methods for the devices regularly used, except SMS that was shown to all participants, as we assumed that they all have at least a feature mobile 4 ).
Moreover, we considered that the device in which the online event triggering a survey was detected could differ from the device in which the invitation to participate was delivered. Consequently, accepting to use a device to trigger in-the-moment surveys in “TriggeringDevice” was not a requirement to use such device to receive invitations. The only exception was Popup-PC. This invitation method, available only in PC, consists in showing a popup window invitation on the participant’s browser when an event of interest is detected in the same PC (for an example, see Revilla & Ochoa, 2018). Therefore, using a popup in a PC requires also to be willing to use a PC to trigger surveys.
As for the question “Method,” we focused on “fast” invitation methods, that is, the ones that show instant notifications to users whenever a new message is received. Thus, the traditional email-PC method was not offered because notifications are not usually a default feature on emails. However, email was considered a suitable messaging system for smartphones and tablets when the participants turn the push notifications on. This requirement was explicit in the question. The order of the invitation methods was randomized.
Finally, the question “FastestMethod” assessed which invitation method participants think they would see first, among those accepted in the question “Method.”
Results
Use of and acceptance to be invited with different devices
99% of respondents declare to regularly use a smartphone, 83% a PC, and 46% a tablet. Both smartphone and tablet show similar levels of use per gender, age, and education groups, while significant differences (5% level) are found for PC, with higher use by males (86.8% versus 79.3%) and more educated panellists (94.6% for high educated, 85.9% for mid-educated, and 71.1% for low educated).
The most used devices are also the most accepted to trigger in-the-moment surveys: 89.1% for smartphone, 86.0% for PC, and 77.3% for tablet (all three significantly different).
Acceptance and coverage of different invitation methods
On average, respondents accept 4.2 invitation methods (69.1% of the methods offered to them). Moreover, 77.6% of them accept three or more invitation methods.
Acceptance and coverage of different invitation methods.
App-smartphone is the most accepted method, followed closely by email-smartphone, while WhatsApp-tablet is the least accepted one. Regarding coverage, as it strongly depends on the level of use of the device, the three smartphone methods are the ones with the greatest levels, followed by SMS and Popup-PC. Although SMS is poorly accepted, it works in any mobile phone, which produces a coverage similar to its acceptance level.
Acceptance and coverage of invitation methods by device and messaging system.
Tablet is the least accepted device to receive invitations, whereas smartphone is the most accepted one (1.3 times more). Such difference, in combination with the level of use of each device, translates into an even larger gap in coverage (2.7 times). Furthermore, WhatsApp is the least accepted messaging system and the app the most accepted one (1.5 times more), whereas the difference in coverage remains proportional as both messaging systems were offered in combination with smartphone and tablet. Finally, as SMS was not linked to a specific device and popup was always linked to PC, their acceptance and coverage remain the same as shown in Table 1.
Combined coverage of different invitation methods
Combined coverage.
If only one method could be offered, it should be app-smartphone. If a second method could be offered, email-smartphone should be chosen as it is the one adding most coverage (i.e., 5.3% of panellists accept this method but do not accept app-smartphone). Popup-PC should be the third method offered (+3.5% coverage) and SMS the fourth (+1.3%). The remaining methods barely add coverage.
Fastest invitation method
Fastest invitation method.
The three invitation methods based on smartphone (the most used device) are the ones that most participants stated they would see first: App-smartphone is the fastest followed by WhatsApp-smartphone, despite being the third least accepted method. This is because WhatsApp is accepted by relatively few panellists, but most of them consider this method the fastest (highest No. First/No. Accept ratio).
Regarding the combined share, if the three smartphone methods were offered, 79.1% of panellists would be using a method that they consider the fastest one. Adding Popup-PC and SMS raises this figure up to 95.9%.
Discussion
In-the-moment surveys can be used to research a variety of topics where memory issues are relevant. However, such surveys suppress one key advantage of web surveys: the possibility for participants of choosing their own schedule for participating (Albaum & Smith, 2012). Consequently, panellists willing to participate may not ultimately do so for several reasons such as lack of time or motivation. But these factors become of secondary importance if participants do not see the survey invitation in time. To that end, fast but acceptable invitation methods should be used.
The current results provide evidence on what invitation methods, among those considered suitable for in-the-moment surveys, are most accepted by panellists of a metered panel in Spain. Receiving invitations in a smartphone using a panel app is the most accepted one. This method is also the one that most panellists consider they would see first. Thus, if only one invitation method is to be used, we recommend using App-smartphone.
However, according to the results of this research, panellists would accept several methods. Thus, when the available time to participate is so critical as in in-the-moment surveys, we recommend that panel companies use multiple invitation methods simultaneously. To select the set of methods offered to panellists, two main aspects should be considered: coverage and speed. Coverage ensures that the maximum number of panellists accept to use one of the offered methods. Speed reduces the risk that panellists who are willing to participate do not see the invitation in time.
Showing invitations by means of an app installed by participants in their smartphones contributes the most to both coverage and speed. That suggests that panels considering conducting in-the-moment surveys should have an app as a starting point, although persuading panellists to install it may be challenging (Revilla, Paura, & Ochoa, 2021). As a second step, encouraging panellists to install an email app on their smartphones (in case they do not have it yet) and activate the push notifications when a new email arrives would significantly contribute also to coverage and speed, without any technological development (panels generally use email as invitation method).
At this point, different directions should be taken to further improve coverage and speed. Regarding coverage, combining smartphone-based methods with one or two alternatives based on PC and/or SMS could be a successful strategy to maximize the coverage of potential participants. Regarding speed, using messaging systems such as WhatsApp should be seriously considered. Although only less than 6 out of 10 participants would accept this method, it is considered as the fastest method by 41% of them (28% of all panellists). However, WhatsApp is currently limiting the maximum number of recipients of a message, even for companies and institutions, which reduces its feasibility as an invitation method.
In any case, offering at least one invitation method per device increases the likelihood of inviting a panellist using the same device where the event of interest is detected, which may contribute to reduce the time needed to see the invitation.
All the above recommendations have in common the need to adapt to participants. Getting people to participate in surveys has been increasingly difficult over the last years (Leeper, 2019). In-the-moment surveys ask panellists to participate under time pressure, which is expected to reduce participation rates. Researchers interested in investigating in the moment should offer new methods to invite panellists, allowing them to choose the more convenient ones. Despite the evidence that combining different invitation methods should be effective in reaching participants in time, overwhelming panellists with repeated messages through different channels could end up with an increased fatigue, leading to panel attrition in the long run. Faced with this dilemma, panel companies need to redesign their incentivization policies to adapt to a reality that is far more complex than rewarding conventional web surveys. Further research is needed to determine the best ways to do so. Finally, the findings presented in this research note are based on a specific sample, country, and kind of in-the-moment surveys. Further research is needed to test the robustness, especially for surveys triggered by different data (e.g., geolocation data), different kinds of samples (e.g., probabilistic panels), and countries. Besides, the reason why some invitation methods are better accepted than others deserves further research. Privacy issues may play a key role in participants’ preferences. Such understanding should allow to design new invitation methods in the future or, at least, communicate the existing ones more effectively.
Footnotes
Acknowledgments
The authors thank Patricia Iglesias for her comments on a previous version of this research note.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No 849165).
