Abstract
Researchers commonly rely on relatively small convenience samples for cognitive pretesting questionnaires. Methods used to recruit these samples vary depending on the population of interest, study timeline, study budget, and other factors. Over the past decade, one method that has become popular because of these considerations is online classified advertisements (e.g., Craigslist ads). A concern with the use of this recruitment method is that it leads to a set of participants who repeatedly participate in cognitive interview studies, changing the cognitive processes used in interviews, potentially resulting in misleading findings. Advertisements placed on social networking sites, such as Facebook, may give researchers more control over targeting recruitment advertisements, produce more participant diversity, and reduce the prevalence of “professional participants” who respond to ads. Recent research has shown that Craigslist and Facebook advertisements do result in selection pools with different demographic characteristics and experiences as study participants. However, we are not familiar with any research that has attempted to address concerns about data quality as a result of (a) professional participant cognitive bias or (b) recruitment method. Using data from two studies for which recruitment advertisements were placed on Craigslist and Facebook, we assess whether there are differences in recruitment speed, demographic diversity, the extent to which professional participants comprise the recruitment pool, and the extent to which a geographically dispersed recruitment pool can be attained. Evidence across the measures of quality was mixed. Facebook advertisements resulted in much faster recruitment than Craigslist advertisements among an online population in which the study topic was virtual worlds and avatars (Study 1), but the inverse was true among an older population in which the study topic was long-term care (Study 2). Mixed evidence was also found for relationships between recruitment platform and demographic composition. In Study 1, we found relationships between recruitment method and education, ethnicity, and race. In Study 2, there was only a relationship between recruitment platform and marital status and employment status. Furthermore, Facebook recruits were significantly younger than Craigslist recruits in Study 1, while in Study 2, Craigslist recruits were younger. Professional participants were identified in the recruitment pools when the concept was operationalized as attempts to deceive the researchers in how they learned about the study. No evidence was found, however, when professional participant was operationalized as the number of times one had participated in research in the past 12 months. Finally, while no comparison is available between platforms, we found that Facebook advertisements resulted in a geographically dispersed recruitment pool with per capita rates ranging from 0.05 to 1.6 and only one state having no representation. The findings from this research will help survey practitioners who conduct cognitive interviews make important decisions in which platforms to expend limited resources for the best recruitment pools from which to draw interview participants.
Keywords
Introduction
Issues With Cognitive Interviews
Over the past 30 years, cognitive interviewing has become one of the prevailing techniques for pretesting survey questionnaires, yet little empirical work has been done to investigate the most effective methods to recruit prospective participants. This is largely attributable to two commonly accepted notions about cognitive interviews. One is that cognitive testing is a qualitative technique limited by the fact that the findings from these interviews are nonrepresentative (e.g., Chan & Pan, 2011). Therefore, at best, practitioners need only to meet quotas for demographic characteristics believed to be important to the study goals (e.g., Blair, Conrad, Ackerman, & Claxton, 2006).
A second reason little attention has been given to how cognitive interview participants are recruited is that modest sample sizes (between 5 and 15) have been considered adequate for finding the measurement errors cognitive interviews are designed to detect (Willis, 2005). Recruiting such modestly sized samples has required relatively little effort. More recently, some scholars have found support for the use of larger samples (Blair & Brick, 2009). There may be good reason for such support. Blair and Conrad (2011), for example, found that relatively large sample sizes were needed to find uncommon issues that would lead to measurement error and a large proportion of problems known to exist in the study test instrument.
Several factors in the field might lead one to expect a body of literature providing empirical evidence on the most effective methods for recruiting prospective study participants. One of these is that researchers often need to recruit to fill quotas of small, disparate, and geographically dispersed groups. Another is the need for larger sample sizes to detect a majority of measurement errors cited previously. A third is the problem of “professional participants,” which has received scant attention in the survey research literature. Despite the need for a body of empirical evidence on effective recruitment methods, there is a dearth of work in this area. In fact, we know of only one conference paper that compared the effectiveness of recruitment methods for cognitive interviews (Murphy et al., 2007). In this article, we identify differences in the effectiveness of various recruitment methods and suggest researchers carefully consider them when designing a cognitive testing study.
Traditional Recruitment Methods
A variety of methods have traditionally been used to recruit cognitive interview participants. Little attention has been given to the extent to which they were effective. Rather, researchers have tended to focus on the mechanics of interviewing. Generally, the only mention about study recruitment comes in a few sentences of the Methods section where authors describe how prospective participants were located and selected.
Commonly used recruitment techniques vary in complexity, reach, and expense, based largely on study needs. Among the commonly used methods are newspaper advertisements (Demaio & Rothgeb, 1996; Nolin & Chandler, 1996; Willis, 2005, Willis et al., 2005), flyers (DeMaio & Rothgeb, 1996; Willis, 1999, 2005; Saleska et al., 2009), intercept methods (Hinsdale, McFarlane, Weger, Schoua-Glusberg, & Kerwin, 2009), recruitment firms or panels (Levin et al., 2009), institutional contacts or personal networks (Willis, 2005), snowball or purposive sampling (Conron & Austin, 2008; Irwin, Varni, Yeatts, & DeWalt, 2009; Miller et al., 2011; Schechter, Blair, & Vande Hey, 1996; Willis, 2005), and even probability sampling such as random digit dialing telephone samples with screening (Caspar & Biemer, 1999; Oksenberg, Cannell, & Kalton, 1991) or sampling from an institutional frame (McCabe, Tanner, & Heiman, 2010).
Online Classified Advertisements
Recently, researchers have begun to use online resources for recruitment. Online classified advertisement sites, the most common of which is Craigslist.com (comScore, 2012), have been used for participant recruitment in studies about diet messaging (Merriam et al., 2009), predicting support for government institutions (Grosskopf & Frye, 2011), question ordering (Van De Walle & Van Ryzin, 2011), developing scales (Palamar, Kiang, & Halkitis, 2011), binge eating (Grilo & White, 2011), sexual violence (Katz-Schiavone, Levenson, & Ackerman, 2008), terrorism (Khurgin-Bott & Farber, 2011), men who have sex with men (Grov, Ventuneac, Rendina, Jimenez, & Parsons, 2013), and recruiting other hard-to-reach populations (Anderson, Wandersee, Arcenas, & Baumgartner, 2013). However, while anecdotally we know that such advertisements are commonly used to recruit for cognitive interview studies, we know of only one conference paper that discusses the use of such advertisements for cognitive interview studies.
Murphy et al. (2007) compared traditional methods of recruitment (newspaper advertisements, flyers, support groups, and personal networks) to online classified advertisements (e.g., Craigslist.com ads). They found evidence that there are demographic differences in the respondents to Craigslist and traditional recruitment advertisements. For example, they found that Craigslist respondents tended to be younger, more likely to identify as an “other” race, and more likely to have a graduate education than respondents to more traditional cognitive interviewing techniques.
Professional Respondents/Participants
It has been observed by some survey practitioners that there is a growing problem with professional participants comprising more of their nonprobability samples (e.g., cognitive testing studies). Professional participants have been defined as those individuals who “[participate] in multiple research studies, sometimes deceitfully, in order to obtain financial rewards or gifts” (Clow & James, 2013, p. 101). Although there seems to be an anecdotal recognition among researchers to the existence and problem of such participants, it has been given little attention in academic journals. The market research literature, on the other hand, recognized the problem long ago. Bogart (1984, p. 82), for example, when speaking of focus groups noted that The trouble is that people who can be enticed into a research laboratory do not always represent a true cross section of potential customers. A cadre of professional respondents are always ready to volunteer …
New methods of recruitment may be preferable for two reasons. Recently developed methods enable researchers to target recruitment to specific demographic groups. They also give researchers the ability to control who is likely to see recruitment advertisements. One new platform on which survey researchers can advertise for nonprobability studies are social media sites (e.g., Facebook).
Today, Facebook is the fourth most visited website, with approximately two thirds of the U.S. adults being Facebook users (comScore, 2012; Rainie, Smith, & Duggan, 2013). With such widespread use, it has become a popular place for advertising. Several noncognitive testing studies, for example, have used Facebook to recruit nonprobability samples (Chu & Snider, 2013; Kapp, Peters, & Oliver, 2013; Lohse, 2013; Morgan, Jorm, & Mackinnon, 2013). These studies have several things in common: (1) they use Facebook to recruit by linking their advertisement to a survey administered online, (2) they have targeted their Facebook advertisements to a specific population, and (3) they focused on the cost-effectiveness of using Facebook advertisements.
Much less attention has been given to other measures of effectiveness or quality. Two exceptions are Fenner et al. (2012) and Ramo and Prochaska (2012). Fenner et al. found that Facebook respondents who completed a screening survey were representative of a target population on age, geographic distribution, and socioeconomic profile. On the other hand, Ramo and Prochaska (2012) found that Facebook respondents, relative to the study target population, were disproportionately male and White. These inconsistent findings suggest that more work is needed to understand the extent to which Facebook advertisements can be used to recruit sample pools that are diverse and reflect the demographics or other characteristics of the target populations or subpopulations.
Hypotheses
We test four hypotheses related to the performance of Facebook versus Craigslist platforms for recruiting cognitive interview participants with regard to the speed of recruitment, diversity of the prospective participants recruited, degree to which recruited prospective participants are professional participants, and extent to which prospective participants are geographically dispersed.
Our first hypothesis tests whether the platforms differ in the rate at which prospective participants are recruited. The platforms have different purposes and audiences that could affect recruitment. Moreover, the platforms might be applied or targeted differently depending on the population of interest for a study. On Facebook, advertising is secondary to the site’s purpose, social networking, but users are passively exposed to ads while they engage with the site. It takes more effort for a user to suppress or not look at the ads than to look at them. Facebook’s advertising feature is designed to let advertisers target populations as specifically and efficiently as possible, by demographic criteria and other personal characteristics of interest. In contrast, there is no other purpose to Craigslist than searching and posting in the classifieds. Craigslist relies on users to be interested, have an idea of what they are looking for, search for opportunities, and click. Craigslist sellers have little control over who sees their ads and must assume interested readers will see their ad while browsing through postings.
More people use Facebook than Craigslist (approximately 140 million monthly U.S. visitors vs. 60 million, according to each website). Additionally, some research suggests that, on average, Craigslist users tend to be younger and better educated than Facebook users (Duggan & Brenner, 2013; Jones, 2009). This is likely due to Facebook’s broader appeal and purpose (social networking vs. classified advertising). These differences in reach and audience demographics could affect the rate at which prospective participants can be recruited from platforms. Another possibility is that a study’s population of interest could affect the rate at which participants are recruited from each platform. A population that is highly represented on one platform, for example, may be more quickly recruited from that platform.
Craigslist and Facebook serve unique and distinct purposes. Craigslist exists to help people buy, sell, and swap goods and services. Users of the website sign on to engage in an exchange of money for goods or services. Respondents to the ads are seeking out ways to make money, perhaps through research. Facebook, on the other hand, has a multitude of uses by individuals, groups, and corporate interests, but is designed as a social networking platform. At its root, it connects people for whatever purpose they have in mind. Respondents to the Facebook ads are likely engaging in a wide variety of social networking behaviors while using the website. A relationship between platform and demographic diversity of prospective participants may be attributable to the platform, but may just as well be attributable to the recruitment technique and tailoring.
Craigslist by design requires localized advertising. It is not designed to be a “global social network,” it is designed to be a local community-based advertising platform. By design, advertisements to Craigslist users are targeted to specific geographic areas. Facebook’s advertising format allows for broader advertising campaigns automatically reaching many different geographic populations at the same time. These differences in how the platforms are intended to be used have the potential to affect the extent to which researchers can use them to recruit dispersed populations.
Method
To test our hypotheses, we conducted two studies, namely, Study 1 (Second Life) and Study 2 (long-term care). The two studies were conducted sequentially, rather than concurrently. The first was an internally funded study to test the feasibility of using new modes of administration to conduct cognitive interviews. The second study was conducted as part of an external project to learn about U.S. adult retirement planning, particularly the extent to which individuals understand and plan to use long-term care insurance. Following is a description of the populations for the two studies, the recruitment techniques used for each study, and the measures included in analyses.
Populations
The target populations of the two studies differed significantly. The target population for Study 1 (Second Life) was residents of the virtual world Second Life. Second Life is one of the largest massively multiplayer online game in the world (Au, 2013). The target population of Study 2 (long-term care) was U.S. adults between the ages of 40 and 75. These populations differ in the technologies they use, how they use them, and the salience the studies may have to them.
Recruitment
Both studies implemented recruitment strategies that included two or more advertising platforms. The primary advertising platforms were Craigslist and Facebook. These platforms have different intended uses, allowable formats for advertisements, and cost requirements. 1
Craigslist is an online community intended to bring people together for a variety of reasons. One significant use of Craigslist is to reach people actively searching for nontraditional opportunities to earn money, such as paid participation in research studies. At the time we undertook this research, there was no charge for most advertisements on Craigslist. Craigslist now charges a nominal fee (US$25) for posting in the “et cetera” section. Facebook, on the other hand, is a for-profit social media company that allows users to connect with friends and family. At the same time, Facebook utilizes user-provided information (e.g., likes, comments, and user-provided demographic and geographic information) to allow companies to target advertisements. Facebook has a number of options for advertising. For these studies, we selected the cost-per-click option in which we identified the characteristics of the target population, set a maximum bid for advertisements, and allowed Facebook to optimize our advertising up to a maximum budget. Across advertisement campaigns, our cost-per-click ranged from US$1.11 to US$1.50. We received 39 clicks for Study 2 advertisements and over 1,800 for Study 1 advertisements.
Facebook and Craigslist also differ in the amount of space allowed for advertisements. Craigslist offers up to 70 characters for advertisement headlines while Facebook allows only 25. In addition, Craigslist does not limit the number of characters for the body of an advertisement whereas Facebook limits it to 15 characters. To mitigate these differences, we kept the themes of advertisements as similar as possible across the platforms. Prospective respondents were offered the same information about the study when they arrived at a screener survey page. Common to four of the five advertisements was a reference to a token incentive (US$50 for the Second Life study and US$40 for the long-term care study).
Study 1 (Second Life)
Advertisements for the Second Life study were run in “waves” for approximately 1 week at a time with 1 week between the first and the second waves. In the first wave, we ran Craigslist advertisements in five cities dispersed across the United States (Raleigh-Durham, North Carolina; New York City, New York; Los Angeles, California; Houston, Texas; and Minneapolis, Minnesota). We chose these cities because they were relatively well dispersed across the country, were reasonably large population centers, and were believed likely to have Second Life users. At the same time, we ran Facebook advertisements targeted at adults age 18 and over who resided in the United States, and “liked” Second Life, avatars, or virtual worlds. 2 The second wave of Craigslist advertisements were run in the same cities as Wave 1, plus San Francisco, Santa Barbara, and San Diego. 3 Targeting criteria for the second wave of Facebook advertisements remained the same. Given the response each type of advertising had in the first two waves, a third wave of advertisements were run only on Facebook to recruit a final set of prospective participants. 4 The images and text of Facebook advertisements were varied as part of a separate experiment to determine if either had an effect on response to Facebook advertisements.
Study 2 (Long-term care)
We released a single wave of advertisements for the long-term care study. Since the long-term care study cognitive interviews were conducted in-person only, we only ran advertisements in the cities where we planned to conduct interviews. Advertisements were placed on Craigslist pages for Raleigh-Durham, North Carolina and Washington, DC pages. Facebook advertisements were targeted to adults between the ages of 40 and 75 who lived in the metropolitan areas of one of the two study cities. 5 An advertisement was also placed on an internal classified page at the researchers’ employer. The site is similar to Craigslist. However, only company employees can view the advertisements. Figure 1 presents examples of Facebook and Craigslist advertisements 6 for both studies.
Facebook example advertisement 1.
All advertisements included a link to a web-administered screening survey. Screening surveys for both studies took approximately 5 min to complete. A unique link was created for each advertisement. The survey software recorded the URL from which respondents were directed to the screener. The combination of these two things allowed us to track responses to each advertisement.
Measures
The screening survey served the purpose of allowing us to identify prospective participants who met study demographic goals, as well as collecting data necessary to evaluate the effectiveness of the different types of advertisements. The variables needed for screening were different across the two studies. For example, marital status was only asked in Study 2 (long-term care). It was needed because marital status can impact retirement decisions. There was no analytic reason to ask this question for Study 1 (Second Life).
Speed of data collection—We tracked the number of responses to each advertisement throughout the time each advertisement was posted. We examined the cumulative frequency completes to each advertisement throughout the advertisements posting. Geographic dispersion— For Study 1 (Second Life), the number of screener completes was tracked by state/territory. To normalize results, we computed a per capita rate by dividing the number of state residents by 100,000 using population statistics from the U.S. Census Bureau
7
and then divided the number of screener completes by this rate. Thus, the measure of geographic dispersion is the number of screener completes per 100,000 residents. Professional participant—As noted previously, some researchers have classified attempts to deceive researchers in an attempt to gain study participation, often with the intent to maximize receipt of financial incentives, as characteristic of professional participants (e.g., Clow & James, 2013). We therefore operationalized professional participant for Study 1 (Second Life), as attempts to mislead researchers about the way in which one learned of the study. This operationalization was used for two primary reasons. One reason is that we believe many professional participants suspect or know such attempts are being monitored and some would attempt to conceal such behavior. Another is that detecting discrepancies in known and stated ways of accessing the screener survey was achievable. To do this, we tracked which advertisement a screener respondent clicked on to arrive at the screener survey, as well as responses to a screener survey question that asked how respondents learned about the study. A dichotomous variable was created for “misleading” the researchers—where there was a discrepancy between the known URL used to reach the screener and the response to the screener question that asked respondents how they learned of the study (1 = attempted to mislead and 0 = no attempt to mislead). In Study 2, a screening question asked respondents how frequently they participated in any research studies in the past 12 months (0, 1–2, 3–4, 5–6, or 7 or more times).
Other measures
We also included demographic measures of age, gender, race, ethnicity, education, income, marital status, and retirement status. A description of these variables is available in Appendix.
Results
Speed of Data Collection
Figure 2 presents the number of screener respondents to Facebook and Craigslist advertisements in the first week of data collection for each study. The x-axis shows the number of days and the y-axis is a count of screener completes on a base-10 logarithm scale. Given the different levels of response across the studies, a log transformation allows all four to be fit on the same graph. Evident is that the two studies had large differences in the number of screener respondents as well as large difference in the advertisement type that was most effective for the study. For example, by the end of the first week of screener data collection Study 1 had 749 screener completes whereas Study 2 had just 4. Conversely, at the end of the first week, Study 2 had 106 Craigslist completes whereas Study 1 had just 20. The null hypothesis of no difference for recruitment speed between platforms was rejected.
Speed of screener completes in first 7 days of each study.
Demographic Characteristics
Again, evidence of demographic differences across the two recruiting methods was mixed. As shown in Table 1, in Study 1 we find evidence of a relationship between recruiting method and education (p < .01), recruiting method and ethnicity (p < .01), and recruiting method and race (p < .01), and a significant difference between the mean age of Craigslist respondents (31.11) and Facebook respondents (41.81; p < .0001, folded f = .1044, pooled t-value = −4.45), but no relationship between recruitment method and gender or income.
Study 1 (Second Life) Screener Survey Demographics.
Note. BA/BS = Bachelor of Arts/Bachelor of Science.
*Fisher’s exact test p < .01. **t-test (Satterthwaite t-value = 2.85, p < .01).
For Study 2, we found no relationship between recruitment method and highest level of education, race, or income. However, as Table 2 shows, we found relationships between advertisement type and marital status (Fisher’s exact test p < .05) and advertisement type and employment status (Fisher’s exact test p < .05). Using an analysis of variance for unbalanced data, 8,9 we also found a statistically significant difference in the mean age of respondents by the type of advertisement to which they responded. The mean age of respondents was Craigslist = 50.32 (SD = 10.29), Facebook respondents = 59.40 (SD = 8.89), friend of family member referral = 65.00 (SD = 7.87), and RTI classified = 37.33 (SD = 6.43). 10
Study 2 (Long-Term Care) Screener Survey Demographics.
Note. BA/BS = Bachelor of Arts/Bachelor of Science.
*Fisher’s exact test p < .05. **Analysis of variance p < .001.
To further investigate the differences in means, we conducted a Tukey’s Studentized Range (honestly significant difference) test. 11 The findings presented in Table 3 suggest there are statistically significant differences (p < .05) in three of the means: Craigslist and family or friend referral (difference = 14.675; confidence interval [CI]: [3.651, 25.699]), Facebook and RTI classifieds (difference = 22.0967; CI [2.833, 41.3]), and family or friend referral and RTI classifieds (difference = 27.667; CI [9.044, 46.920]).
Study 2 (Long-Term Care) Tukey’s Studentized Range (HSD) Post Hoc Test Comparison of Mean Ages.
Note. CI = confidence interval; HSD = honestly significant difference.
*p < .05.
Professional Participants
Evidence for professional participants seeking to participate in our studies was mixed. In Study 1, the findings suggest that professional participants attempted to mislead researchers in order to gain participation in the study. For example, 32.1% of the screener respondents who arrived at the screening survey through Craigslist reported of learning about the study in a way different than the way the researchers knew had learned about it. Approximately 6.5% of the Facebook respondents did the same. We calculated a t-test (Satterthwaite t-value = 2.85, p < .001) of the differences in misreporting means and found the difference in misreporting across advertisement types to be statistically significant (p < .0001, F-value = 3.74). That these were intentional attempts to mislead the researchers is further evidenced by the fact that seven of the nine Craigslist respondents who misreported how they learned of the study indicated they heard of the study from sources on which no advertisements were placed (e.g., Second Life classifieds [n = 4] and a newspaper or blog ad [n = 2]).
On the other hand, using a different operationalization, we found no evidence of professional participants in the Study 2. For example, Table 4 shows a Fisher’s exact test 12 of the association between frequency of research study participation in the past 12 months and the type of advertisements prospective participants responded to. Using this test, we find no statistically significant relationship between the two.
Study 2 (Retirement Planning) Fisher’s Exact Test to Detect Association Between Advertisement Type and Frequency of Participation in Research in the Past 12 Months.
p = .8463.
Geographic Dispersion
As seen in Figure 3, there was a widespread interest in participating in Study 1 cognitive interviews. We had only one state that was not represented in the Facebook sample for Study 1 (North Dakota). However, no obvious pattern is evident in the data. The top five states per capita (100,000) were Vermont (1.60), Oregon (0.92), Wyoming (0.87), West Virginia (0.81), and Washington (0.78). The bottom five states/territories per capita were Puerto Rico (0.05), the District of Columbia (0.14), Mississippi (0.20), South Carolina (0.30), and Utah (0.32). The harmonic mean for per capita rates was 0.40 per 100,000. 13
Study 1—Second Life heat map plotting completes per 100,000 residents.
Discussion and Conclusion
Most previous research on cognitive interviewing has focused on the mechanics of conducting the interview. Little has been done to evaluate recruitment methods. An exception is work done by Murphy et al. (2007) comparing Craigslist to more traditional methods (e.g., print advertisements). But, changes in communication and popular media in the past few years have provided new avenues for recruitment. Facebook, in particular, has gained a significant number of users. In the process, it has become a significant avenue for advertising of all types.
Several recent studies have been done to evaluate Facebook for study recruitment. Yet, they have focused on the extent to which Facebook could be used to recruit a study population or the cost-effectiveness of Facebook advertising (e.g., Chu & Snider, 2013; Kapp et al., 2013; Lohse, 2013; Morgan et al., 2013). Declining budgets for social research and recent findings showing larger samples may be needed to find a large proportion of measurement errors in questionnaires (Blair & Conrad, 2011) suggest more understanding is needed of how the use of recruitment platforms affects participant pools.
Overall, our findings indicate the population of interest has a significant role in determining which platform produces the best recruitment results. For example, in Study 1 (Second Life), Facebook was clearly the faster and more effective recruitment technique. It not only drew more screener completes, it did so at a much faster pace. For Study 2 (the retirement study), the opposite was true. Craigslist drew a much larger pool of prospective participants and did so much more quickly. One possible explanation for these findings is how the populations use the two platforms. In some of our cognitive interviews, we learned that Second Life users use Facebook to interact with other users. They maintain friendships developed “in-world” outside the virtual world environment. Put another way, Facebook seemed to be a place where Second Life users continued to be engaged with the virtual community. None of our retiree study participants mentioned using Facebook to seek retirement information. The difference in engagement between the two populations likely played a role in their willingness to click on advertisements for studies targeted at groups related to those factors.
For demographic diversity, the only consistent finding was that participants recruited through Craigslist were significantly younger than participants recruited through Facebook. Given the 2009 Pew data referenced previously, in which Craigslist is found to be more popular with younger, better educated people, it is not surprising that significantly younger participants were recruited through Craigslist for the Second Life study. It is an interesting finding, however, that Craigslist resulted in younger recruits for the retirement study. One possible explanation is those nearing retirement may be seeking new places to both earn additional income and find new activities. Because of Craigslist’s intended use it offers individuals a place to do both.
We also found mixed evidence for a relationship between recruitment platform and inclusion of professional participants in recruitment pools. That is, in the Second Life study, we recruited a larger number of professional participants from Craigslist than Facebook. We found little evidence of professional respondents in the observations from either platform on the retirement study.
We think this finding is likely explained by how we operationalized professional participant in the two studies. We agree with Clow and James (2013) professional participants will attempt to deceive researchers in order to gain financial rewards. So, in the Second Life study, we determined whether recruits were professional participants, whether there was a discrepancy between the way they reported getting to the recruitment screener survey, and how we knew they got there. By using this operationalization, we assume two things. First, we assume professional participants know social research and marketing professionals have a preference for nonprofessional participants (e.g., those who rarely participate in research) and are attempting to deceive researchers. Second, we assume they know professional participants are overrepresented on Craigslist. Professional participants therefore have an incentive to mislead researchers about how they arrive at a recruitment site. In the retirement study, we operationalized professional participant using a question about the number of times one has participated in research in the recent past. We did this, in part, as an attempt to find another way of operationalizing the concept. In hindsight, we think this may have exposed the purpose of the question to professional participants.
Limited data are available to test the hypothesis for geographic dispersion. Clearly, greater population dispersion occurred from Facebook recruiting, even though the theoretical sizes of the study population who could view the advertisements were comparable. One explanation for this is that while a comparable number of people could theoretically see the advertisement on Craigslist that did not happen.
Caveats and Suggestions for Future Research
Our findings must be interpreted cautiously. This study did not include a randomized, controlled sample of participants and we do not have a direct comparison with traditional, non-Internet methods of recruiting. However, the results should be helpful in defining questions for future research. Specifically, it will be important to determine whether there are other populations for which Facebook would be so effective or whether there are unique characteristics to the Second Life user that drove the findings of the current research. The use of Facebook to recruit other online-specific populations that perhaps are not so rare sites may be evaluated as a comparison.
Another important question is if the platforms are functionally equivalent enough to compare and evaluate. It is clear that they serve two different purposes, but as Facebook becomes ever more commercial and advertisement driven, will the profit motive drive all users’ interactions? Another question to be answered by further research is what wording, image, and layout manipulations result in the most effective recruitment ads? Finally, perhaps the most important question for survey practitioners is what impact, if any, recruitment platforms have on cognitive interview results. Future research should examine metrics of cognitive interview quality (e.g., number of errors identified and number of changes recommended) by platform of recruitment to determine whether there are meaningful differences in quality as a result of how individuals are recruited. For example, are there differences in cognitive interview data provided by professional and nonprofessional participants?
Footnotes
Appendix
Gender—dichotomous variable (female = 1). Race—categorical variable including White, Black or African American, American Indian or Alaska Native, Asian, Hawaiian or other Pacific Islander, and other. Ethnicity—dichotomous variable for Hispanic or non-Hispanic. Education—education was measured by the highest degree or year of school completed for each study. However, the categories were slightly different for the two studies. For Study 1 (Second Life), the education categories were less than high school, high school diploma, some college no degree, associate degree, bachelor’s degree (BA/BS), and postbaccalaureate degree. The education categories for Study 2 (long-term care) were less than high school, high school diploma, vocational/technical/ or trade school degree, some college no degree, bachelor’s degree (BA/BS), and postbaccalaureate degree. Income—measured as a categorical variable for both studies. Due to different study needs, the studies used different scales. Study 1 (Second Life) had the following six categories: US$20,000 or less, US$20,001–US$40,000, US$40,001–US$60,000, US$60,001–US$80,000, US$80,000–US$100,000, and more than US$100,000. Study 2 (long-term care) had only the following three categories: less than US$35,000, US$35,000–US$74,999, and US$75,000 or more. Marital status—a dichotomous variable for married or not married. This variable was only used in Study 2 (long-term care). Retirement status—dichotomous variable for retired or not retired. This variable was also only used in Study 2 (long-term care). Age—age in years.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.