Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany

Background, Scope, and Research Questions

Non-Internet-based survey panels have been around for many years (e.g., the Dutch Telepanel; Saris, 1998). They served as a blueprint for various large-scale panel infrastructures that have been established recently, allowing researchers to collect probability-based survey data (Blom et al., 2016; Bosnjak, Das, & Lynn, 2016; Hays, Liu, & Kapteyn, 2015). Examples are the LISS Panel in the Netherlands (Das, 2012; Scherpenzeel & Das, 2011; Scherpenzeel & Toepoel, 2012), the ELIPSS Panel in France (Revilla, Cornilleau, Cousteaux, Legleye, & de Pedraza, 2015), and the Understanding America Study panel in the United States (Hays et al., 2015). The overall aim of all these infrastructures is to enable researchers in the social, behavioral, and health sciences to collect high-quality survey data, using cross-sectional or longitudinal survey designs.

Although similar infrastructures have been created in Germany by commercial vendors or by academic institutions operating as data collection platforms for a limited set of eligible researchers pursuing specific research topics (e.g., in case of the German Internet Panel; Blom, Gathmann, & Krieger, 2015), there was a lack of an open probability-based panel for Germany until 2013.

Open probability-based panels share three common characteristics (Das, Kapteyn, & Bosnjak, 2017): (1) openness in terms of being accessible for academic researchers from any substantive area to field primary studies and to use the data collected, (2) probability -based and therefore optimized for yielding accurate population estimates in the respective countries, and (3) transparency in terms of the processes by which these infrastructures have been built and are being operated. In addition, the data collection process and its deliverables are transparent, facilitating replicability of processes and outcomes. To fill the gap of having no such research infrastructure available to the academic research community in Germany until 2013 fulfilling all these three characteristics, the GESIS Panel was created.

The GESIS Panel is a probability-based mixed-mode panel infrastructure operated by GESIS—Leibniz Institute for the Social Sciences in Mannheim, Germany. It offers the academic research community a unique opportunity to collect survey data from a probability-based sample of the German population. To compensate for noncoverage among non-Internet users (e.g., de Leeuw, 2005) and to include all persons not willing to participate in online surveys into the panel, each bimonthly GESIS Panel data collection wave is administered in two self-administered survey modes, namely (1) an online mode through web-based surveys and (2) an off-line mode through paper-and-pencil surveys sent via postal mail.

The processes pursued to recruit panelists to set up the GESIS Panel, and the extent to which the specific mixed-mode strategy employed in the GESIS Panel yielded satisfactory levels of sample composition bias over key demographic variables, are the two major topics addressed in this report. Following Chang and Krosnick (2009), the term sample composition bias denotes the deviation of (panel-based) sample characteristics in relation to established national benchmarks such as the Census in the United States. In our case, the equivalent national benchmark capturing the German general population is the German Microcensus (GMC; http://www.gesis.org/en/services/data-analysis/official-microdata/), which is available for the same year the GESIS Panel was built (2013). Specifically, this report seeks to provide answers to three research questions:

First, how dissimilar is the GESIS Panel sample in relation to the German general population? To assess the degree of sample composition bias, we refer to the GMC and will compare the degree of dissimilarity for variables which have been collected both in the GESIS Panel as well as in the GMC.

Our second research question asks if the inclusion of respondents not using the Internet, and those panelists not willing to participate online despite having Internet access, improves the overall sample composition in relation to the GMC compared to an online-only panel strategy? Answers to this question help to determine if, to what extent, and for which variables the mixed-mode strategy used in the GESIS Panel has been advantageous.

Our third research question asks if the dissimilarity between the GESIS Panel and the GMC is in line with what can be expected from other general population surveys, such as the German General Social Survey ALLBUS (Blohm & Koch, 2015) and the German part of the European Social Survey (ESS; Koch, Halbherr, Stoop, & Kappelhof, 2014). To provide answers to this research question, we will compare dissimilarity metrics across three pairs: GESIS Panel versus GMC, ALLBUS versus GMC, and ESS versus GMC.

Before we summarize the findings on the three research questions addressed, we describe the process of building the GESIS Panel in the next section.

Sampling Procedure

The main objective of the sampling strategy used for the GESIS Panel was to provide a probability sample of the target population. That implies known inclusion probabilities of all elements in the population. The theoretical basis of probability-based methods allows inferring from a sample to the target population (Cochran, 1977). The target population for the GESIS Panel includes all German-speaking persons aged between 18 and 70 residing in private households that are registered in Germany (i.e., born between December 1, 1942, and November 30, 1995). The goal of the GESIS Panel was to recruit about 4,000 panelists. To achieve this goal, the fieldwork agency was instructed to conduct face-to-face interviews using a gross sample from the local population registers of German municipalities (cf. fieldwork report by Steinacker & Schmidt, 2014). Therefore, the sample frame is defined as the union of all population registers of the municipalities within the Federal Republic of Germany excluding municipalities on islands, which are inaccessible by car. A two-stage sampling procedure was applied to realize the sampling strategy: sampling of municipalities and sampling of individuals.

Firstly, municipalities as primary sampling units (PSUs) were selected using a stratified random sampling approach (Lohr, 2010, Chapter 3). The federal states of Germany, the administrative districts, and the settlement structure were used to form the strata. In total, 270 sample points were allocated to the strata proportionally to the size of the target population. The allocation, and hence the complete sampling of municipalities, is based on population estimations as well as the territorial status before the results of the 2011 German census were available (Steinacker & Schmidt, 2014). Within each stratum, the sample points were selected by proportional-to-size random sampling. Since some large municipalities have been drawn more than once, the selected sample points are distributed over 236 different municipalities.

Secondly, individuals were selected from the sampled PSUs. The fieldwork agency ordered 140 addresses of individuals that were randomly sampled from local population registers of the sampled municipalities. In total, 81 addresses within each sample point were drawn from the supplied pool of addresses using a systematic random sampling approach.

Out of the 236 municipalities, 10 refused to provide addresses. Five of them were replaced by structurally similar municipalities from the corresponding stratum. For the remaining 5 municipalities, the sampling of individuals was carried out by an alternative approach (address-listing approach using telephone registers). Hence, the gross sample included 21,870 individual members.

Recruitment Procedure

The GESIS Panel employed a multistage procedure to recruit respondents (see Figure 1 for an overview) that stretched over a period of 6 months.

OPEN IN VIEWER

Figure 1.

The multistage recruitment process of the GESIS Panel.

At Stage 0 of the recruitment process, the sampling frame was defined as all German-speaking persons aged between 18 and 70 residing in private households that are registered in Germany, from which a probability sample of individuals was drawn (Stage 1 in Figure 1, encompassing 21,870 addresses of individuals). Next, face-to-face (computer-aided personal interviews) recruitment interviews with 7,599 respondents were conducted (Stage 2 in Figure 1). Those having expressed their willingness to participate in the GESIS Panel (6,210 interviewees; Stage 3 in Figure 1) were invited to a self-administered profile survey (online survey and paper questionnaire). Participation in the profile survey was mandatory to become a member of the GESIS Panel, yielding an initial starting sample encompassing 4,938 panelists (Stage 4 in Figure 1).

Recruitment Interview

All gross sample members were first contacted through a short prenotification letter that contained information about the study and about the conditional €5 incentive for completion of the personal interview, and a data protection leaflet. Shortly after the prenotification letter, respondents were contacted personally by an interviewer. In case of noncontact, interviewers left a postcard informing about their visit, while a minimum of four contact attempts was required. Among other measures to increase response, a telephone hotline was provided by the field institute where contacted sample members could receive additional information about the study. Overall, 267 interviewers worked on the GESIS Panel recruitment. Detailed information about the fieldwork of the recruitment interview can be found in Steinacker and Schmidt (2014).

To keep the burden for the respondents low, the interview was kept short: The median duration was 15 min. The survey was presented as a multitopic survey of general interest, and any questions were avoided that could induce a topic-related bias (for details, refer to the recruitment report by Schaurer, Struminskaya, & Enderle, 2014). The recruitment survey contained questions on well-being, trust, leisure activities and Internet use, affinity towards technology, survey experience, and several sociodemographic questions. The interview ended with the question on the willingness to participate in future self-administered surveys of about 20-min duration administered every 2 months. The interviewer provided information about the GESIS Panel including the information on the €5 incentive for the participation in each self-administered wave and handed over a GESIS Panel leaflet. Self-administered surveys for which respondents were recruited could be completed using online or paper questionnaires. Respondents who expressed their willingness to participate in the panel were assigned to the online or the off-line mode. Respondents who used the Internet were assigned to the online mode. Interviewers were asked to present online participation as an attractive option and to persuade respondents to participate in the online mode, but Internet users were also free to opt for paper questionnaires. Those respondents who did not use the Internet were assigned to the off-line mode. In the remainder of this article, we refer to the first group as online respondents and to the second group as off-line respondents.

Respondents who agreed to participate in the panel in the online mode were asked to provide their e-mail addresses. For collecting an e-mail address from the respondent, interviewers received an incentive additional to their standard remuneration. A further incentive was paid to the interviewer if a respondent participated in the first self-administered survey, independently of the survey mode.

The response rate for the personal interview (Stage 1) was 35.5% (AAPOR RR1) resulting in 7,599 interviews. With respect to Stage 2, in the recruitment interview, 81.72% of respondents were willing to participate in future surveys, yielding a recruitment rate of 28.98% (based on AAPOR RR1). Figure 2 summarizes the different stages and the sample growth across time during the recruitment process.

OPEN IN VIEWER

Figure 2.

Sample growth summary across time during the recruitment process for the GESIS Panel.

Self-Administered Profile Survey

Overall, 6,210 respondents of the recruitment interview expressed their willingness to participate in future self-administered surveys. Both online and off-line respondents were mailed an invitation letter for the profile survey within 2 weeks after their participation in the recruitment interview. ¹

The invitation letter included an unconditional incentive of €5 in cash. For online respondents, it provided a link to the online survey and an access code. For off-line respondents in addition to the invitation, a paper questionnaire and a return envelope were enclosed. Off-line respondents who wanted to switch to the online mode could do so using the URL pointing to the online survey and the login information provided in the letter. Both groups received a postal reminder 1 week after the invitation letter. The reminder was sent out independently of actual participation and was framed as a thank you and a reminder letter. A second reminder was sent via e-mail to those respondents of the online group who had not finished the survey about 2 weeks after the first invitation. The e-mail reminder was sent to the respondents who provided a valid e-mail address. The profile survey contained questions from sociology and political science such as questions on media use, political attitudes, quality of life, leisure activities, attitudes about work, and employment. The profile survey was designed to be enjoyable and not burdensome for the respondents in order to facilitate further panel participation. Additionally, it served as a training survey for respondents so that they could familiarize themselves with the procedure of the survey invitation and survey answering process.

Completion of the online survey or returning the paper questionnaire was a precondition of becoming a panel member, that is, to be considered a part of the active panel as defined by Callegaro and DiSogra (2008). The response rate to the profile survey (or profile rate, DiSogra & Callegaro, 2016) is 79.42% overall, 78.78% for the online mode and 80.67% for the off-line mode. The cumulative response rate for the profile survey at the start of the GESIS Panel in 2014 was 23.02% (Schaurer et al., 2014, p. 9). ²

Regular Data Collection Waves

The regular GESIS Panel survey waves take place bimonthly, each encompassing about 20 min and having initially started mid-August 2013. Each fielding phase lasts 2 months. The first three waves in 2013 were part of the recruitment phase and therefore only included a subsample of panelists (see Figure 2). Those waves were implemented to avoid a long break after the profile survey without contact to respondents and comply with the bimonthly survey pattern announced during the recruitment interview. These three initial waves differ from the regular waves regarding the incentive procedure, the invitation mode, and response rates. In those waves in 2013, respondents were incentivized with conditional incentives of €5 for administrative reasons and online respondents received the invitation only by e-mail.

The regular panel process with the full sample started in February 2014. From then onward, the GESIS Panel schedules six waves annually. In these regular panel waves, all respondents are invited by mail with an unconditional incentive of €5. The invitation letter for online participants refers to an upcoming invitation e-mail, which includes the URL to the online survey. The invitation letter for off-line participants is accompanied with the paper questionnaire and a prepaid return envelope. Only online participants receive a first reminder after 1 week of fielding time and a second reminder after 2 weeks of fielding time. Panelists who do not participate in the GESIS Panel for three consecutive waves either due to noncontact or due to refusal are excluded from the panel and do not receive any further invitations. An in-house panel manager serves as a contact person for respondents and answers all requests. The panel management is also responsible for updating addresses of returned invitation letters.

Like the LISS Panel, the GESIS Panel encompasses longitudinal core studies and studies submitted by researchers from various disciplines such as sociology, psychology, political science, and economy. Submitted studies can be either cross-sectional or longitudinal with repeated measures across several waves and usually do not exceed 5 min of survey time within each wave questionnaire. By early 2017, the completion rates per wave were about 90% of invited respondents for the online mode and about 85% for the off-line mode.

Sample Composition Bias Operationalization

Regarding sample composition bias, the proportions for key demographic variables in the GESIS Panel are compared to the GMC as benchmark for the German population. Corresponding comparisons with the GMC can also be computed for established probability-based surveys such as the German General Social Survey (ALLBUS; Blohm & Koch, 2015) and the German part of the ESS (Koch et al., 2014).

To operationalize the degree of dissimilarity between corresponding variable distributions, the Duncan dissimilarity index (Duncan & Duncan, 1955) is used, measuring the deviation of the sample composition from a respective benchmark presumed to represent the overall population composition. Because of its appropriateness to operationalize the degree of mismatch (e.g., Knudsen & Fortheringham, 1986), the Duncan index has been used in various areas including survey-related applications (e.g., Blumenstiel & Gummer, 2015; Lipps, Pekari, & Roberts, 2015).

The Duncan dissimilarity index of a variable is calculated as the absolute difference between the proportions of the sample and the benchmark in Category c, summed over all categories of a variable, divided by two and multiplied by 100:

where C is the number of categories and p_c is the proportion of Category c. The interpretation of the index is straightforward since it reports the percentage of cases in the sample that must switch categories in order to have the same marginal distribution as the benchmark. For instance, a dissimilarity index amounting to 5 indicates that 5% of subjects would need to be reclassified to other categories to match a corresponding reference distribution. The closer the value of the index is to 0, the more similar the marginal distributions are. For the computation of standard errors and 99% confidence intervals (CI), stratified bootstrapping with sampling points as strata has been used (see Shao & Tu, 1995; Sitter, 1992).

Sample composition bias across recruitment stages

Table 1 summarizes the dissimilarity of the different samples across the recruitment stages of the GESIS Panel depicted in Figure 1 in comparison to the GMC 2013. The overlap between corresponding variables is summarized across eight dimensions that are available in both surveys (gender, age, citizenship, marital status, household size, place of birth, education, household income) for this comparison.

OPEN IN VIEWER

Table 1.

Duncan Dissimilarity Indices of the GESIS Panel Across Recruitment Stages in Comparison to the German Microcensus 2013.

Dimension	Variable	Stage 2: Recruitment Interview	Stage 3: Profile Survey	Stage 4: Initial Panel
(1)	Gender	0.67	1.32	1.89
		[−0.59, 1.93]	[−0.26, 2.89]	[0.18, 3.59]
(2)	Age	2.06	2.23	3.44
		[1.01, 3.11]	[0.91, 3.55]	[1.85, 5.03]
(3)	Citizenship	2.52	3.52	4.97
		[1.76, 3.28]	[2.70, 4.34]	[4.16, 5.79]
(4)	Marital status	3.67	1.88	4.60
		[2.34, 5.00]	[0.41, 3.36]	[2.83, 6.38]
(5)	Household size	4.72	4.80	4.36
		[3.62, 5.82]	[3.42, 6.19]	[3.11, 5.61]
(6)	Place of birth	1.97	3.42	5.07
		[0.98, 2.96]	[2.30, 4.54]	[3.92, 6.23]
(7)	Education	7.05	8.71	9.81
		[5.81, 8.29]	[7.30, 10.13]	[7.30, 10.13]
(8)	Household income	17.68	14.96	13.89
		[16.49, 18.87]	[13.6, 16.32]	[12.34, 15.44]
(1–8)	Mean dissimilarity across all eight comparison dimensions	5.04 [4.64, 5.44]	5.11 [4.63, 5.58]	6.01 [5.46, 6.55]

Across the different recruitment stages, the mean dissimilarity indices tend to get larger (from D = 5.04 in Stage 2 to 6.01 in Stage 4). While the CIs between subsequent recruitment stages overlap, suggesting nonsignificant changes, the contrast between Stages 2 and 4 indicate a significantly more pronounced mean sample composition bias.

Among the eight demographic variables, the degree of mismatch with the GMC increases significantly for citizenship (more German citizens in the GESIS Panel) and place of birth (more German-born participants in the GESIS Panel). However, for household income, Stage 4 (D = 13.89) indicates a significantly smaller sample composition bias compared to Stage 2 (D = 17.68).

Overall, these findings suggest that sample composition bias remained constant across recruitment stages for five (gender, age, marital status, household size, education) out of eight variables, worsened for two variables (citizenship, place of birth), and improved for the household income estimate.

GESIS Panel mode differences in terms of sample composition

Table 2 summarizes the dissimilarity in comparison to the GMC 2013 for the entire starting sample of the GESIS Panel (stage 4), and for the subset of initial panel members participating online only.

OPEN IN VIEWER

Table 2.

Duncan Dissimilarity Indices of the Initial Panel (Stage 4 According to Figure 1) Between Modes.

Dimension	Variable	GESIS Panel Online Participants Only	GESIS Panel Full Sample, Encompassing Both Online and Off-Line Participants
(1)	Gender	1.08	1.89
		[−0.85, 3.00]	[0.18, 3.59]
(2)	Age	4.86	3.44
		[2.80, 6.92]	[1.85, 5.03]
(3)	Citizenship	5.22	4.97
		[4.22, 6.22]	[4.16, 5.79]
(4)	Marital status	2.55	4.60
		[1.16, 3.95]	[2.83, 6.38]
(5)	Household size	7.01	4.36
		[5.07, 8.94]	[3.11, 5.61]
(6)	Place of birth	5.98	5.07
		[4.61, 7.36]	[3.92, 6.23]
(7)	Education	21.36	9.81
		[19.43, 23.30]	[8.24, 11.38]
(8)	Household income	14.79	13.89
		[12.83, 16.75]	[12.34, 15.44]
(1–8)	Mean dissimilarity across eight comparison dimensions	7.86 [7.15, 8.57]	6.01 [5.46, 6.55]

Overall, the mean dissimilarity index computed across all eight corresponding comparison dimensions reported in Table 2 indicates that the inclusion of off-line respondents, and those panelists not willing to participate online despite having private Internet access, improved the overall similarity of the sample in comparison to the GMC significantly. When analysing the specific changes on a variable level it becomes evident that this overall improvement is decisively influenced by the reduction in dissimilarity on the education variable. In other words, while an online-only strategy would have yielded a fairly accurate representation of the GMC for the demographic variables gender, age, citizenship, marital status, household size, place of birth, and household income, the mixed-mode strategy of the GESIS Panel has contributed to improve the representation of the GMC on education decisively.

Sample composition bias comparisons between probability-based surveys

For the following comparisons, the intersecting set of variables is just five instead of eight. The variables gender, age, citizenship, marital status, and household size have been measured correspondingly in all surveys considered. However, details about place of birth, education, and household income are not reported in the respective publications (ALLBUS: Blohm & Koch, 2015; ESS: Koch et al., 2014).

For each of the five demographic variables considered, the following four dissimilarity indices have been computed: GESIS Panel recruitment interview respondents versus the GMC, GESIS Panel initial panel members versus GMC, ALLBUS 2008 versus GMC, and ESS 2010 versus GMC. In addition, Table 3 reports mean dissimilarity indices for all comparisons across the five corresponding comparison dimensions.

OPEN IN VIEWER

Table 3.

Duncan Dissimilarity Indices of the GESIS Panel, ALLBUS, and ESS Compared to the German Microcensus.

Dimension	Variable	GESIS Panel 2013 Versus GMC 2013		ALLBUS 2008 versus GMC=2008	ESS Round 5 2010 Versus GMC 2010
		Recruitment Interview	Initial Panel
(1)	Gender	0.67	1.89	0.40	2.80
		[−0.59, 1.93]	[0.18, 3.59]
(2)	Age	2.06	3.44	3.10	5.90
		[1.01, 3.11]	[1.85, 5.03]
(3)	Citizenship	2.52	4.97	1.90	3.30
		[1.76, 3.28]	[4.16, 5.79]
(4)	Marital status	3.67	4.60	4.50	1.30
		[2.34, 5.00]	[2.83, 6.38]
(5)	Household size	4.72	4.36	2.95	6.20
		[3.62, 5.82]	[3.11, 5.61]
(1–5)	Mean dissimilarity across all five comparison dimensions	2.73 [2.20, 3.25]	3.85 [3.08, 4.62]	2.57	3.90

Note. ESS = European Social Survey; GMC = German Microcensus. Columns 2 and 3: own calculations; Column 4: ALLBUS 2008 versus GMC 2008: Blohm and Koch (2015, p. 108–110); Column 5: ESS Round 5 versus GMC: Koch et al. (2014, p. 16).

The overall mean dissimilarity index computed across the five variables gender, age, citizenship, marital status, and household size for the GESIS Panel recruitment interview (D = 2.73; 99% CI [2.2, 3.25]) is not significantly different from the dissimilarity of ALLBUS 2008 and the GMC (D = 2.57), and significantly lower than the one estimated for the ESS-GMC comparison (D = 3.90). Table 3 also summarizes the discrepancy scores for each demographic variable separately, indicating that the largest match between the GESIS Panel recruitment interview sample and the GMC has been achieved for gender (D = 0.67; 99% CI [−0.59, 1.93]), and the largest discrepancy was found for household size (D = 4.72; 99% CI [3.62, 5.82]).

The initial GESIS Panel sample represents the benchmark, aggregated across the five demographic variables considered (D = 3.85; 99% CI [3.08, 4.62]), equally well as ESS Round 5 (D = 3.90), and shows a slightly larger overall dissimilarity score in comparison to the ALLBUS 2008 (D = 2.57). The lowest discrepancy score was found for gender (D = 1.89; 99% CI [0.18, 3.59]), the largest for citizenship (D = 4.97; 99% CI [4.16, 5.79]).

Overall, these findings suggest that the dissimilarity between the GESIS Panel and the GMC is fully in line with what can be expected from other probability-based general population surveys such as the German General Social Survey ALLBUS and the German part of the ESS.