Measurement Reliability,Validity,and Quality of Slider Versus Radio Button Scales in an Online Probability-Based Panel in Norway

Abstract

Little is known about the reliability and validity in web surveys, although this is crucial information to evaluate how accurate the results might be and/or to correct for measurement errors. In particular, there are few studies based on probability-based samples for web surveys, looking at web-specific response scales and considering the impact of having smartphone respondents. In this article, we start filling these gaps by estimating the measurement quality of sliders compared to radio button scales controlling for the device respondents used. We conducted therefore two multitrait–multimethod (MTMM) experiments in the Norwegian Citizen Panel (NCP), a probability-based online panel. Overall, we find that if smartphone respondents represent a nonnegligible part of the whole sample, offering the response options in form of a slider or a radio button scale leads to a quite similar measurement quality. This means that sliders could be used more often without harming the data quality. Besides, if there are no smartphone respondents, we find that sliders can also be used, but that the marker should be placed initially in the middle rather than on the left side. However, in practice, there is no need to shift from radio buttons to sliders since the quality is not highly improved by providing sliders.

Keywords

measurement quality reliability and validity slider scales radio buttons web surveys probability-based online panel multitrait–multimethod experiment

The increase in the number of web surveys has been accompanied by an increase in the literature studying the comparability of the data coming from web surveys with the one coming from more traditional modes of data collection, in terms of representativeness and of data quality (Cobanoglu, Warde, & Moreo, 2001; Dillman, 2000; Dillman et al., 2008; Funke, 2016; Heerwegh, 2009; Kreuter, Presser, & Tourangeau, 2008), as well as specific challenges and opportunities of web surveys (see for instance, Ilieva, Baron, & Healy, 2001).

Different studies compared typical web scales to more traditional scales. Some study drag-and-drop scales (e.g., Kunz, 2015; Sikkel, Steenbergen, & Gras, 2014), drop-down menus (Christian, Dillman, & Smyth, 2007; Couper, Tourangeau, Conrad, & Crawford, 2004; Leeuw, Hox, & Dillman, 2008; Liu & Conrad, 2016), or order-by-click scales (e.g., Revilla, Ochoa, & Loewe, 2013). In this article, we are interested in sliders scales, that is, that the respondent shall answer by moving a marker along a line to rate the item.

Previous research suggests that sliders are not the preferable form of response scales. Sikkel, Steenbergen, and Gras (2014) found that slider scales are perceived more enjoyable than traditional radio button scales by respondents but only at the first-time answering this kind of scale. The use of sliders is more cognitively demanding and hence completion time, break offs, and missing answer are increasing when sliders are used instead of radio button scales (Cook, Heath, Thompson, & Thompson, 2001; Couper et al., 2004; Funke, Reips, & Thomas, 2011; Husser & Fernandez, 2013; Roster, Lucianetti, & Albaum, 2015; Sikkel et al., 2014). Buskirk, Saunders, and Michaud (2015), in contrast, found that completion time for sliders starting at the middle or the right is shorter but causes higher item nonresponse than radio button scales. If the starting point of the slider is on the scale, then it is not clear whether a respondent indeed would choose that position and therefore does not move the slider or whether he or she refuses to answer. Sellers (2013) also found that the initial position of the slider marker has an effect on the response: Markers positioned initially at the high point of the scale yield higher scores, and those at the low point of the scale yield lower scores. Cape (2009) found that slider scales produced more accurate responses than other scales, while Couper, Tourangeau, Conrad, and Singer (2006) did not find a difference in the response distribution compared to other scales such as radio button scales. Finally, in their study, Buskirk and Andrus (2014) saw no differences between PC and smartphones regarding different aspects (recruitment, survey completion, and items), whereas Buskirk et al. (2015) results suggested that slider scales are more often preferred by mobile respondents but may not be as much as radio button scales. Overall, it is often recommended to stick to radio button scales or other more typical web scales instead of using sliders.

However, there are several reasons why it continues to be of interest to study the use of sliders. First, scales that a few years ago could seem hard to use or led to technical problems (and thereby likely to frustration and breaking off the survey) may perform much better nowadays because the capabilities of PCs and mobile devices and the experience of people in using such devices have increased vastly.

Second, as pointed out by Couper and Miller (2008, p. 831), “a key characteristic of Web surveys is their diversity.” Indeed, web surveys vary a lot at many levels: the topics, the kind of recruitment of the respondents and of samples, the optimization or not of this layout for mobile devices, and so on. It is very difficult to make general conclusions about the performance of response scales in web surveys based on the small body of existing literature.

Third, there is no previous research estimating the measurement reliability and validity, the complements of random and systematic errors, of these scales. However, having information about the measurement reliability and validity is crucial for several reasons: It helps designing new questionnaires (Revilla, Zavala-Rojas, & Saris, 2016; Saris & Gallhofer, 2014) and allows correcting for measurement error, which is a necessary step to get accurate conclusions (DeCastellarnau & Saris, 2014; Saris & Revilla, 2016).

Few multitrait–multimethod (MTMM) experiments have already been conducted in web surveys. Overall, this research suggests that the quality of web survey questions is quite high and similar to the one in more traditional modes like face-to-face (Decastellarnau & Revilla, 2017; Revilla, 2010, 2015; Revilla & Saris, 2013a, 2015). Nevertheless, the quality varies across languages, topics, and scales. Thus, this research is still too limited to make general inferences that could really inform web survey designers and/or to allow correction for measurement errors in web surveys. There is therefore a clear need of more evidence about the reliability and validity of survey questions in the case of web surveys.

The goal of this article is to start filling in these gaps by estimating the reliability, validity, and quality of sliders compared to radio button scales using two MTMM experiments implemented in the Norwegian Citizen Panel (NCP), a probability-based online panel in Norway. The NCP allows the respondents to answer the survey on their preferred device, on a PC, tablet, or smartphone. Hence, we can estimate the reliability, validity, and quality only for PC/tablet respondents (excluding smartphone respondents) and for the whole set of respondents (including smartphones) to see whether the presence of smartphone respondents affects the quality because both the smaller screens and the touch screen navigation are likely to provoke differences.

The remainder of this article is divided as follows: in the second section, we give further information about the data, and in the third section, we present the MTMM approach, the experiments, and the analyses more in details. In the forth section, we provide then the main results, and finally, in the fifth section, we conclude.

Data

Data Collection in the Norwegian Citizen Panel Wave 7

The NCP is a research-purpose Internet panel with more than 10,000 panel members. A probability-based sample of the Norwegian population from 18 to 95 years old was drawn from the Norwegian National Registry. Panel members complete an online questionnaire of about 20 min twice a year. As an incentive, a lottery on a travel gift card valued 25,000 NOK (about 2,650EUR) is included in each round. In general, the NCP does not provide explicit “Don’t Know” options, but respondents are not forced to answer whether they do not want. More information about the NCP can be found at http://digsscore.uib.no/methodology

Wave 7 took place during the month of November and the 2 first days of December 2016. After the invitation e-mail and three reminders, the overall response rate was 72%. A total of 4,689 panelists participated in this wave. Participants were divided in four subsets that received different questions. Respondents could participate using the device they wanted to answer the survey and the NCP recorded this information as a variable. Ideally, we would have analyzed separately all three types of devices: PCs, tablets, and smartphones. However, the NCP just differentiated between smartphone respondents and nonsmartphone respondents. Twenty-six percent of all panelists participated in the questionnaire using a smartphone and 74% used either a tablet or a PC (Skjervheim & Høgestøl, 2016). Comparing the sociodemographic characteristics of these two groups, we only find significant differences in gender and age: Men and young respond more on a smartphone (see Supplemental Online Material A).

The percentage of mobile respondents that did not complete the survey to such an extent that they were classified as nonrespondents was more double (8.7%) than for PC/tablet respondents (4.2%). This means that smartphone respondents were more likely to leave the survey before the end of the questionnaire. Unfortunately, we cannot test for a possible selective nonresponse bias since we do not have data of those excluded. Moreover, analyzing smartphone respondents alone would have led to very small sample sizes, and we can therefore just compare the results for PC/tablet respondents only and for all respondents together (PC/tablet and smartphones).

Method

The MTMM True Score (TS) Model

The MTMM approach (Andrews, 1984; Campbell & Fiske, 1959) consists in repeating a set of questions measuring correlated concepts of interest (called traits), using several methods, for instance, several scales (radio button, drag-and-drop, and slider). This approach has provided a large amount of information about the measurement quality, as product of reliability and validity, of survey questions for many different topics, countries, languages, and response scale formats (Alwin, 1997; Andrews, 1984; Revilla & Ochoa, 2015; Revilla, Saris, & Krosnick, 2014; Rodgers, Andrews, & Herzog, 1992; Saris & Gallhofer, 2007, 2014; Saris, Revilla, Krosnick, & Shaeffer, 2010; Scherpenzeel & Saris, 1997).

Different models have been proposed to analyze MTMM experiments but using the TS model proposed by Saris and Andrews (1991) appears to be the most attractive option because (1) this model provides better fit compared to others (Corten et al., 2002; Saris & Aalberts, 2003) and (2) it allows estimating the reliability and validity of survey questions separately, as can be seen in the system of equations below or in the graphical representation (see Supplemental Online Material B).

Y_{i j} = r_{i j} T_{i j} + e_{i j},

T_{i j} = v_{i j} F_{i} + m_{i j} M_{j},

where F_i is the i^th trait or factor, M_j is the j^th method, Y_ij is the observed answer for the i^th trait and the j^th method, T_ij is the TS or systematic component of the response, r_ij is the reliability coefficient, v_ij is the validity coefficient, and e_ij is the random error associated with Y_ij.

Equation (1) defines each observed variable as the sum of the associated systematic component and random errors. Equation (2) indicates that each systematic component is itself the sum of the trait component and the effect of the method used to assess it. Then, the total measurement quality is obtained by taking the product of the reliability and validity: q²_ij = r²_ij × v²_ij.

In addition, in the initial TS-MTMM model, we assume that (1) the random errors are uncorrelated with each other and with the independent variables in the different equations, (2) the traits are correlated, (3) the method factors are uncorrelated between them and with the traits, and (4) the impact of the method factor on the traits measured with a common scale is the same.

The Split-Ballot TS-MTMM

To be identified, such a TS-MTMM model usually requires including at least three correlated traits, each measured with three different methods. In order to reduce the cognitive burden of the respondents and to limit possible memory effects due to the repetitions of the same questions to the same respondents (van Meurs & Saris, 1990), Saris, Satorra, and Coenders (2004) proposed to combine the MTMM approach with a split-ballot approach: Respondents are split randomly into several groups; each split-ballot group gets a combination of two methods for a given set of three traits, instead of getting all the three methods. Two designs are possible: The two-group design which has the advantage that all respondents get Method 1 (M₁) when answering the questions the first time and then half of the sample gets Method 2 (M₂) and the other half Method 3 (M₃) when answering the questions a second time. The MTMM model is still identified under quite general conditions when a split-ballot design is used (Saris, Satorra, & Coenders, 2004). However, in practice, nonconvergence problems and improper solutions occur frequently for the two-group design (Revilla & Saris, 2013b). In order to reduce these problems, in this study, we implemented the three-group split-ballot design: Respondents are randomly assigned to three groups: Group 1 gets Methods 1 and 2, Group 2 gets Methods 2 and 3, and Group 3 gets Methods 3 and 1.

The Two MTMM Experiments in the NCP Wave 7

Different experiments were included in Wave 7. In one of the four subsets of the sample, two three-group split-ballot MTMM experiments were implemented to evaluate the reliability and validity of a set of six traits. This set of experimental questions was answered by a subset of 1,199 panelists divided in three groups (372 in Group 1, 410 in Group 2, and 417 in Group 3) following the split-ballot design described just before. The average nonresponse rate for those questions was 1.5% and the proportion of smartphone respondents around 35%. The first experiment asked about the immigration in Norway (“qualification of immigrants” experiment), and the second one asked about the Norwegian Supreme Court (“evaluation of court” experiment). Table 1 presents the exact wording for each trait in each experiment.

Table 1.

The Three Requests for Answers Analyzed for Each Experiment.

Experiment	Requests for Answers
Qualifications: Immigrants	How important do you think… t₁ =…having good educational qualifications should be in deciding whether someone born, brought up, and living outside Norway should be able to come and live here? t₂ =…coming from a Christian background should be in deciding whether someone should be able to come and live here? t₃ =…having work skills that Norway needs should be in deciding whether someone should be able to come and live here?
Evaluation: Court	To what extent do you think… t₁ =…the Norwegian Supreme Court’s power to decide certain types of controversial issues should be reduced? t₂ =…the decisions of the Norwegian Supreme Court favor certain groups more than others? t₃ =…that if the Norwegian Supreme Court starts to make several decisions that most people disagree with, the institution might just as well be dissolved?

Experiment

Requests for Answers

Qualifications: Immigrants

How important do you think…

t₁ =…having good educational qualifications should be in deciding whether someone born, brought up, and living outside Norway should be able to come and live here?

t₂ =…coming from a Christian background should be in deciding whether someone should be able to come and live here?

t₃ =…having work skills that Norway needs should be in deciding whether someone should be able to come and live here?

Evaluation: Court

To what extent do you think…

t₁ =…the Norwegian Supreme Court’s power to decide certain types of controversial issues should be reduced?

t₂ =…the decisions of the Norwegian Supreme Court favor certain groups more than others?

t₃ =…that if the Norwegian Supreme Court starts to make several decisions that most people disagree with, the institution might just as well be dissolved?

Each request for answer was measured with three different methods, which are presented in Table 2. The methods are similar in both experiments. Group 1 always answered M₁ first and later M₂, whereas Group 2 answered M₂ and later M₃ and Group 3 answered M₃ and later M₁.

Table 2.

The Three Methods Analyzed: Main Characteristics and Visual Layout for PC/Tablet Respondents.

M ₁	Rating scale with five radio buttons from not at all to completely
M ₂	Slider continuous scale—51 points—from not at all to completely starting at the middle
M ₃	Slider continuous scale—51 points—from not at all to completely starting at 1

All three methods use partially labeled scales where the end points are not at all or completely. However, M₁ is a rating scale with five radio buttons, whereas M₂ and M₃ are slider scales with underlying 51 points, that is, not visible to the respondent. In M₂, the marker is initially placed in the middle of the scale, whereas in M₃ it is placed on the left side.

As the sliders allow for a higher differentiation of the answers, we expect that the quality for M₁ will be lower than the one for M₂ and M₃, taking advantage of the sliders’ properties to present a continuous scale.

As the marker is initially placed on the left side of the slider, we expect that the means and distributions will be shifted towards the left when asking the questions using M₃ instead of M₂ where the marker is initially place in the middle. Hence, we expect more measurement errors when the marker is initially placed on the left (more systematic errors).

Smartphone respondents got a different visual representation of the questions in order to enhance their experience. The radio button scales were vertical, and the sliders layout was changed (see Supplemental Online Material C).

Analyses and Testing

The split-ballot TS-MTMM models were estimated for each experiment using the maximum likelihood multiple-group estimation procedure of LISREL (Jöreskog & Sörbom, 1996) with the Pearson correlation matrices, means, and standard deviations as input data, for an example of the initial LISREL input, see Supplemental Online Material D. The groups correspond to the different split-ballot groups. The models are analyzed first for PC/tablet respondents only and then for all respondents together (PC/tablet and smartphones).

Before looking at the estimates, each model is carefully tested for misspecifications following the procedure proposed by Saris, Satorra, and van der Veld (2009), which is implemented in the JRule 3.0.4 software (van der Veld, Saris, & Satorra, 2009). A misspecification is defined as a deviation larger than .4 for the standardized loadings and than .1 for the causal effects and correlations (default values of the software). If a parameter is or not misspecified, it is identified by using information about the power of the test, the modification indices, and the expected parameter change.

Starting from the initial model described before, the model is corrected step by step when misspecifications are found, till an acceptable fit is achieved, for a list of the extra parameters freed and information about the fit of the final model used see Supplemental Online Material E.

Since we used a three-group split-ballot MTMM design, we expect differences in quality depending if the method is asked at the beginning or at the end of the survey. Respondents can learn how to answer using specific response formats, giving more accurate answers at the end of the survey (in this case, the quality will increase) or they can get tired of answering (in that case the quality will decrease), so even if, as a starting point, we assumed that the quality for a given method is similar in the different split-ballot groups, when necessary we allowed variations.

Results

Preliminary Analyses

Before getting into the main results, we did some preliminary analyses to test whether the means and distributions of the questions included in the MTMM analyses differ: (a) across time, that is, the same method asked at the beginning or at the end and (b) across methods, that is, two different methods at each point in time. Differences could be attributed to the split ballot design: Different respondents for the same method at different points in time and different respondents for the different methods at a given point in time, besides time and method. However, this is not the case: There are no significant differences between groups on the key demographic variables age, gender, municipal size, and education (see Supplemental Online Material F).

In order to make both types of scales, the categorical radio button scale and the continuous slider scale, comparable in terms of means and distributions, we converted both slider scales following two different approaches: (1) to compare distributions, we converted both continuous scales to categorical scales with five categories: 0–10, 11–20, 21–30, 31–40, and 41–50, and (2) to compare means, we used a typical rescaling method: Values in between a 0 and 50 range were transformed to values in between a 1 and 5 range using the following formula:

x_{t} = 4 (x_{i} - x_{min / max} - x_{min}) + 1,

where x_min and x_max are the minimum and maximum possible rating in the specific scale for the item and x_i is the item rating.

These preliminary analyses consider all respondents together (PC, tablet, and smartphone respondents). The methods for which we found significant differences (5% level) across time are presented in Table 3.

Table 3.

Methods for Which We Found Significant Differences Across Time.

Experiment	Request	Differences in distributions	Differences in means
Qualifications: Immigrants	t ₁	M₂ and M₃	M ₂
	t ₂	M ₃	M ₃
	t ₃	M₂ and M₃	M ₂
Evaluation: Court	t ₁	M ₃	M₁ and M₂
	t ₂	M₂ and M₃	M ₂
	t ₃	M ₃	M ₃

There are often significant differences across time both in terms of distributions and means for the two slider scales, whereas for the radio button scale, only one significant difference is found (for the first request of the “evaluation of the court” experiment, in terms of means). Exploring the distributions, we find that for the sliders, less respondents selected the initial marker position when answering at the end, possibly increasing the accuracy of the answers. The significant differences across methods at a given point in time are presented in Table 4.

Table 4.

Significant Differences Across Methods at a Given Point in Time.

Experiment	Time	Request	Differences in distributions	Differences in means
Qualifications: Immigrants	B	t ₁	M₃ vs. M₂; M₁ vs. M₃	M₃ vs. M₂; M₁ vs. M₂
		t ₂	M₃ vs. M₂; M₁ vs. M₃	M₃ vs. M₂; M₁ vs. M₃
		t ₃	All	All
	E	t ₁	M₁ vs. M₃	M₁ vs. M₂; M₁ vs. M₃
		t ₂	M₃ vs. M₂; M₁ vs. M₂	M₁ vs. M₂
		t ₃
Evaluation: Court	B	t ₁	M₃ vs. M₂; M₁ vs. M₃	All
		t ₂	All	All
		t ₃	M₃ vs. M₂; M₁ vs. M₃	M₃ vs. M₂; M₁ vs. M₃
	E	t ₁	M₃ vs. M₂	M₃ vs. M₂; M₁ vs. M₂
		t ₂	All	M₃ vs. M₂; M₁ vs. M₂
		t ₃	M₃ vs. M₂	M₃ vs. M₂; M₁ vs. M₂

Note: “All” means that all three tests (M₁ vs. M₂, M₁ vs. M₃, and M₂ vs. M₃) lead to significant differences. B = beginning; E = end.

At a given point in time, significant differences in distributions and in means are generally found across methods. Thus, overall, we can conclude that the use of different scales to measure the same concepts leads to different distributions and means and that asking the same request using the same scale at different points in the survey also leads to different distributions and means for slider scales. The next question is: are reliability, validity, and quality different too?

Reliability, Validity, and Quality Estimates

Analyzing the two split-ballot MTMM experiments implemented in the NCP in the way described before, we obtained estimates of the reliability, validity, and quality of each request for an answer when asked with each of the three methods. Table 5 presents these estimates for each experiment, request for an answer and method, first for PC/tablet respondents only and then for all respondents. When the estimates vary depending on the time of the method, both are indicated: The estimates when the method is at the beginning (B) or at the end (E).

Table 5.

Reliability, Validity, and Quality Estimates for the Different Traits, Methods, Experiments for PC/Tablet Only or for All Devices Together.

Experiment	Devices Included	Method/Time	Reliability				Validity				Quality
Experiment	Devices Included	Method/Time	t ₁	t ₂	t ₃	Avg	t ₁	t ₂	t ₃	Avg	t ₁	t ₂	t ₃	Avg
Qualification: Immigrants	PC and tablet	M ₁	.79	.83	.77	.80	.94	.94	.94	.94	.75	.79	.73	.75
		M ₂	.83	.88	.86	.86	.98	.98	.98	.98	.81	.87	.85	.84
		M ₃	.81	.88	.83	.84	.96	.94	.96	.95	.78	.83	.80	.80
	All	M ₁	.83	.83	.77	.81	.96	.96	.96	.96	.80	.80	.74	.78
		M ₂	.81	.83	.81	.82	.96	.94	.96	.95	.78	.78	.78	.78
		M ₃	.81	.85	.83	.83	.94	.92	.94	.93	.76	.78	.78	.77
Evaluation: Court	PC and tablet	M₁/B	.67	.76	.69	.71	.96	.96	.96	.96	.65	.73	.66	.68
		M₂/B	.69	.74	.81	.75	.90	.90	.98	.93	.62	.67	.79	.69
		M ₃	.72	.77	.69	.73	.74	.77	.94	.82	.53	.60	.65	.59
		M₁/E	.85	.76	.69	.77	.96	.96	.96	.96	.81	.73	.66	.73
		M₂/E	.83	.74	.81	.79	.99	.90	.98	.93	.75	.67	.79	.74
	All	M₁/B	.62	.77	.69	.70	.94	.94	.96	.95	.59	.73	.66	.66
		M₂/B	.67	.76	.79	.74	.85	.85	.96	.88	.57	.64	.76	.66
		M ₃	.66	.76	.74	.72	.74	.79	.94	.82	.49	.60	.70	.59
		M₁/E	.79	.77	.69	.75	.94	.94	.96	.94	.75	.73	.66	.71
		M₂/E	.85	.76	.79	.80	.85	.85	.96	.88	.72	.64	.76	.71

Note. B = beginning; E = end; M₁ = radio button scale; M₂ = slider starting with the marker in the middle; M₃ = slider with the marker starting on the left. The bold values are significant at 0.05.

First, Table 5 shows that the data quality is overall quite high in both experiments, for all requests for an answer and methods. The reliability is usually lower than the validity, sometimes quite a lot. To simplify the discussion, and because we are interested in comparing the quality across methods, we will focus for the rest of the discussion on a comparison of the averages across all three requests for an answer (column “Avg”). In order to assess the significance of the differences between the average coefficients, we conducted several two-tailed z tests for two means. Table 6 presents the significant differences between methods across devices (PC/tablets or all) and Table 7 presents the significant differences between devices across methods.

Table 6.

Significant Differences Between Methods of the Quality Estimates for PC/Tablet Only and for All Devices Together.

Experiment	Devices Included	Significant Differences
Qualifications: Immigrants	PC and tablets	All
Qualifications: Immigrants	All	None
Evaluation: Court	PC and tablets	M₁/B vs. M₃; M₃ vs. M₁/E; M₃ vs. M₂/E
Evaluation: Court	All	None

Note: “All” means that all three tests (M₁ vs. M₂, M₁ vs. M₃, and M₂ vs. M₃) lead to significant differences. B = beginning; E = end.

Table 7.

Significant Differences of Quality Estimates Between Devices Across Methods.

Experiment	Significant Differences
Qualifications: Immigrants	M ₂
Evaluation: Court	None

Since normally researchers are only asking once the questions and differences are not statistically significant, to simplify the discussion, we are mainly interested in comparing the estimates of the methods at the beginning. Considering all respondents, independently of the device they used to answer the survey, using the radio button (M₁) or the slider with the marker starting at the middle point (M₂) leads to a similar measurement quality, in both experiments. In addition, the slider with the marker starting on the left side (M₃) leads to a similar quality estimate in the “qualification of immigrants” experiment, but not in the “evaluation of the court” experiment. Even though, differences between methods when considering all the respondents are not statistically significant.

If instead we consider only the PC/tablet respondents, there are significant divergences between methods: The slider with the marker at the middle (M₂) is the one with the highest quality in both experiments. However, the order of the two others (radio button and slider with the marker on the left) differs. On the one hand, in the “qualification of immigrants” experiment, while the quality of the radio button scale decreases a little bit, the quality of the two sliders increases somehow, leading to a higher quality estimate for M₂, followed by M₃ and with the lowest quality for M₁. On the other hand, for the “evaluation of the court” experiment, the quality increases a little for M₁ and M₂ whereas it stays the same for M₃, increasing the differences between, on one side, M₁ and M₂, and on the other side, M₃.

Focusing on the reliability and validity estimates may help to understand those differences. For instance, looking at the “evaluation of the court” experiment, we can see that M₁ and M₂ have equal reliability estimates independently of the device used, so quality differences are explained by their validity coefficients, which are much lower for M₃.

Getting in the detail of how adding the smartphones respondents affects reliability and validity, we notice that in the “evaluation of the court” experiment most of the decrease is observed for the validity. Moreover, it seems that this is mainly because of M₂. Furthermore, for the “qualification of immigrants” experiment, M₂ also seems to be more responsive to adding smartphone respondents, but in this case the decrease in reliability is slightly bigger. Looking at Table 7, we see that the only significant difference between coefficients when adding the smartphone respondents is the one from Method 2 for the “qualification of immigrants” experiment.

Conclusions

In this article, our goal has been estimating the measurement reliability and validity of different questions asked using different scales in a probability-based online panel in Norway (the NCP). We have focused on a comparison of the quality in radio button versus slider scales and have considered the impact of having smartphone respondents and not only PC/tablet respondents since previous research on these aspects was scarce.

The data came from two split-ballot MTMM experiments implemented in the seventh wave of the NCP, which dealt with the qualification of immigrants and the evaluation of the Supreme Court. In each experiment, three different requests for an answer were asked each using three different methods: a radio button, a slider with the marker initially placed on the middle, and a slider with the marker initially placed on the left.

Overall, when considering the whole sample (i.e., all respondents, independently of the device they used to complete the survey), we have found that the measurement quality in the NCP is quite high for all traits and methods, in both experiments (.72 on average). It is quite comparable to what is found in previous research. The validity is in general higher than the reliability, sometimes quite a lot. On the contrary, we have found similar quality levels for the radio button and the slider with the marker starting on the middle, on both experiments, which differs from what we expected. When considering all the respondents the differences between the quality estimates of all three methods are not statistically significant, which implies that the three methods can be considered equally advisable.

When considering only PC and tablet respondents, more variations are observed across the methods and if the slider with the marker in the middle is the one with the highest quality in both experiments, the order between the two other methods changes across experiments. Our results show that when including smartphone respondents, the average measurement quality decreases, but differences are not statistically significant, which suggests that the measurement quality for smartphone respondents is not necessarily lower. Furthermore, considering that the differences between methods are not statistically significant when adding the smartphone respondents, we can point out that previous research not including smartphone respondents and/or not separating smartphone from PC/tablet respondents may not be generalizable to studies where the smartphone respondents represent a large part of the sample or even the whole sample.

However, we should keep in mind that these results have limits. In particular, they are only based on: (a) Norway, in which the Internet coverage is much higher than in the rest of the world; (b) one probability-based panel, the NCP, which might differ from other online panels; and (c) two specific topics, which are quite different the one form the other but might lead to different results of what other topics would do. Therefore, further research should test the robustness of the results.

In addition, because of the small sample size, we could not analyze the smartphone respondents by themselves but only deduced what the measurement reliability and validity could be based on the results of PC/tablets and of all respondents together. For a similar reason, we also could not separate PC and tablet respondents to see whether differences exist between these two device types. Further research in that direction would be interesting.

Finally, even if the research has limitations that should be kept in mind, this new study can help to set some practical recommendations on topics where little was known. If smartphone respondents represent a nonnegligible part of the whole sample, our results suggest that using sliders or radio button scales leads to an overall quite similar measurement quality for different questions and topics. In practice, it means that sliders could be used more often without harming the data quality. Besides, if there are no smartphone respondents, sliders can be used, but placing the marker initially in the middle should be preferred to placing it on the left side. However, radio buttons being often simpler to implement, we can also say that there seems to be no need to change them for sliders since the quality is not highly improved by providing sliders. Nevertheless, we should remark that we only studied here the measurement quality and that other aspects could be affected, for instance the dropout rate, item nonresponse or the overall satisfaction of the respondents with the completion of the survey.

Supplemental Material

Supplementary_Material - Measurement Reliability, Validity, and Quality of Slider Versus Radio Button Scales in an Online Probability-Based Panel in Norway

Supplementary_Material for Measurement Reliability, Validity, and Quality of Slider Versus Radio Button Scales in an Online Probability-Based Panel in Norway by Oriol J. Bosch, Melanie Revilla, Anna DeCastellarnau, and Wiebke Weber in Social Science Computer Review

Footnotes

Authors’ Note

We are thankful to the Norwegian Citizen Panel team to accept our proposal and for their help in setting up the experiment.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Alwin

D. F.

(1997). Feeling thermometers versus 7-point scales. Sociological Methods & Research, 25, 318–340. Retrieved from https://doi.org/10.1177/0049124197025003003

Andrews

F. M.

(1984). Construct validity and error of survey measures: A structural modeling approach. Public Opinion Quarterly, 48, 409–442.

Buskirk

T. D.

Andrus

C. H.

(2014). Making mobile browser surveys smarter: Results from a randomized experiment comparing online surveys completed via computer or smartphone. Field Methods, 26, 322–342. Retrieved from https://doi.org/10.1177/1525822X14526146

Buskirk

T. D.

Saunders

Michaud

(2015). Are sliders too slick for surveys? An experiment comparing slider and radio button scales for smartphone, tablet and computer based surveys. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology, 9, 229–260.

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105. Retrieved from https://doi.org/10.1037/h0046016

Cape

(2009). Slider scales in online surveys. CASRO Panel Conference. New Orleans, LA. Retrieved from http://www.websm.org/uploadi/editor/doc/1436872048Cape_2009_Slider_Scales_in_Online_Surveys.pdf

Christian

L. M.

Dillman

D. A.

Smyth

J. D.

(2007). Helping respondents get it right the first time: The influence of words, symbols, and graphics in Web surveys. Public Opinion Quarterly, 71, 113–125. Retrieved from https://doi.org/10.1093/poq/nfl039

Cobanoglu

Warde

Moreo

P. J.

(2001). A comparison of mail, fax and web-based survey methods. International Journal of Market Research, 43, 405–410.

Cook

Heath

Thompson

R. L.

Thompson

(2001). Score reliability in webor internet-based surveys: Unnumbered graphic rating scales versus likert-type scales. Educational and Psychological Measurement, 61, 697–706. Retrieved from https://doi.org/10.1177/00131640121971356

10.

Corten

I. W.

Saris

W. E.

Coenders

van der Veld

W. M.

Aalberts

C. E.

Kornelis

(2002). Fit of different models for multitrait–multimethod experiments. Structural Equation Modeling: A Multidisciplinary Journal, 9, 213–232. Retrieved from https://doi.org/10.1207/S15328007SEM0902_4

11.

Couper

M. P.

Miller

P. V.

(2008). Web survey methods: Introduction. Public Opinion Quarterly, 72, 831–835. Retrieved from https://doi.org/10.1093/poq/nfn066

12.

Couper

M. P.

Tourangeau

Conrad

F. G.

Crawford

S. D.

(2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22, 111–127. Retrieved from https://doi.org/10.1177/0894439303256555

13.

Couper

M. P.

Tourangeau

Conrad

F. G.

Singer

(2006). Evaluating the effectiveness of visual analog scales: A web experiment. Social Science Computer Review, 24, 227–245. Retrieved from https://doi.org/10.1177/0894439305281503

14.

DeCastellarnau

Revilla

(2017). Two approaches to evaluate measurement quality in online surveys: An application using the Norwegian citizen panel. Survey Research Methods, 11(4), 415–433. https://doi.org/10.18148/srm/2017.v11i4.7226

15.

DeCastellarnau

Saris

W. E.

(2014). A simple way to correct for measurement errors in survey research. Retrieved from http://essedunet.nsd.uib.no/cms/topics/measurement/

16.

de Leeuw

E. D.

Hox

Dillman

(Eds.) (2008). International handbook of survey methodology. New York, NY: Taylor & Francis. Retrieved from https://doi.org/10.4324/9780203843123

17.

Dillman

D. A.

(2000). Mail and internet surveys: The tailored design method. New York, NY: John Wiley.

18.

Dillman

D. A.

Phelps

Tortora

Swift

Kohrell

Berck

Messer

B. L.

(2008). Response rate and measurement in mixed mode surveys using mail, telephone, interactive voice response, and internet. Social Science Research, 38, 1–48.

19.

Funke

(2016). A web experiment showing negative effects of slider scales compared to visual analogue scales and radio button scales. Social Science Computer Review, 34, 244–254. Retrieved from https://doi.org/10.1177/0894439315575477

20.

Funke

Reips

Thomas

R. K.

(2011). Sliders for the smart: Type of rating scale on the web interacts with educational level. Social Science Computer Review, 29, 221–231. Retrieved from https://doi.org/10.1177/0894439310376896

21.

Heerwegh

(2009). Mode differences between face-o-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21, 111–121. Retrieved from https://doi.org/doi:10.1093/ijpor/edn054

22.

Husser

J. A.

Fernandez

K. E.

(2013). Survey practice. Survey Practice, 6. Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/40

23.

Ilieva

Baron

Healy

N. M.

(2001). On-line surveys in international marketing research: Pros and cons. International Business, 44, 1–32.

24.

Jöreskog

K. G.

Sörbom

(1996). LISREL 8: User’s reference guide. Uppsala, Sweden: Scientific Software International.

25.

Kreuter

Presser

Tourangeau

(2008). Social desireability bias in CATI, IVR, and web surveys: The effects of modeand question sensibility. Public Opinion Quarterly, 72, 847–865. Retrieved from https://doi.org/10.1093/poq/nfn063

26.

Kunz

(2015). Rating scales in web surveys: A test of new drag-and-drop rating procedures—WebSM. Retrieved May 23, 2017, from http://tuprints.ulb.tu-darmstadt.de/5151/7/Kunz_2015_Rating_scales_in_web_surveys.pdf

27.

Liu

Conrad

F. G.

(2016). An experiment testing six formats of 101-point rating scales. Computers in Human Behavior, 55, 364–371. Retrieved from https://doi.org/10.1016/j.chb.2015.09.036

28.

Revilla

(2010). Quality in unimode and mixed-mode designs: A Multitrait-Multimethod approach. Survey Research Methods, 4(3), 151–164.

29.

Revilla

(2015). Comparison of the quality estimates in a mixed-mode and a unimode design: an experiment from the European Social Survey. Quality and Quantity, 49(3), 1219–1238. https://doi.org/10.1007/s11135-014-0044-5

30.

Revilla

Ochoa

(2015). Quality of different scales in an online survey in Mexico and Colombia. Journal of Politics in Latin America, 7, 157–177.

31.

Revilla

Ochoa

Loewe

(2013). How the form of asking respondents’ three preferred brands changes the results. Metodologia de Encuestas, 16, 67–82.

32.

Revilla

Saris

W. E.

(2013a). A Comparison of the Quality of Questions in a Face-to-face and a Web Survey. International Journal of Public Opinion, 25(2), 419–432.

33.

Revilla

Saris

W. E.

(2013b). The split-ballot multitrait–multimethod approach: Implementation and problems. Structural Equation Modeling: A Multidisciplinary Journal, 20, 27–46.

34.

Revilla

Saris

W. E.

(2015). Estimating and comparing the quality of different scales of an online survey using an MTMM approach. In Engel

(Ed.), Survey measurement: techniques, data quality and sources of error (pp. 53–74). Frankfurt, New York.

35.

Revilla

Saris

W. E.

Krosnick

J. A.

(2014). Choosing the number of categories in agree–disagree scales. Sociological Methods & Research, 43, 73–97. Retrieved from https://doi.org/10.1177/0049124113509605

36.

Revilla

Zavala-Rojas

Saris

W. E.

(2016). Creating a good question: How to use cumulative experience. In Wolf

Joye

Smith

T. W.

(Eds.), The SAGE-handbook of survey methodology (pp. 236–254). London, UK: Sage.

37.

Rodgers

Andrews

Herzog

(1992). Quality of survey measures: A structural modeling approach. Jounal of Official Statistics, 8, 251–275.

38.

Roster

C. A.

Lucianetti

Albaum

(2015). Exploring slider vs. categorical response formats in web-based surveys. Journal of Research Practice, 11, 1.

39.

Saris

W. E.

Aalberts

(2003). Different explanations for correlated disturbance terms in MTMM studies. Structural Equation Modeling: A Multidisciplinary Journal, 10, 193–213. Retrieved from https://doi.org/10.1207/S15328007SEM1002_2

40.

Saris

W. E.

Andrews

F. M.

(1991). Evaluation of measurement instruments using a structural modeling approach. In Biemer

P. P.

Groves

R. M.

Lyberg

L. E.

Mathiowetz

N. A.

Sudman

(Eds.), Measurement errors in surveys (pp. 575–598). New York: John Wiley and Sons, Inc.

41.

Saris

W. E.

Gallhofer

I. N.

(2007). Design, evaluation, and analysis of questionnaires for survey research. Hoboken, NJ: John Wiley.

42.

Saris

W. E.

Gallhofer

I. N.

(2014). Design, evaluation, and analysis of questionnaires for survey research (2nd ed.). Hoboken, NJ: John Wiley. Retrieved from https://doi.org/http://dx.doi.org/10.1002/9780470165195

43.

Saris

W. E.

Revilla

(2016). Correction for measurement errors in survey research: Necessary and possible. Social Indicators Research, 127, 1005–1020. Retrieved from https://doi.org/10.1007/s11205-015-1002-x

44.

Saris

W. E.

Revilla

Krosnick

J. A.

Shaeffer

E. M.

(2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4, 61–79. Retrieved from https://doi.org/10.18148/srm/2010.v4i1.2682

45.

Saris

W. E.

Satorra

Coenders

(2004). A new approach to evaluating the quality of measurement instruments: The split-ballot MTMM design. Sociological Methodology, 34, 311–347.

46.

Saris

W. E.

Satorra

van der Veld

W. M.

(2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling: A Multidisciplinary Journal, 16, 561–582. Retrieved from https://doi.org/10.1080/10705510903203433

47.

Scherpenzeel

A. C.

Saris

W. E.

(1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25, 341–383. Retrieved from https://doi.org/10.1177/0049124197025003004

48.

Sellers

(2013). How sliders bias survey data. MRA’s Alert!, 53, 56–57.

49.

Sikkel

Steenbergen

Gras

(2014). Clicking vs. dragging: Different uses of the mouse and their implications for online surveys. Public Opinion Quarterly, 78, 177–190. Retrieved from https://doi.org/10.1093/poq/nft077

50.

Skjervheim

Ø.

Høgestøl

(2016). Norwegian Citizen Panel 2016, Seventh Wave Methodology report. Retrieved from https://scholar.uib.no/sites/default/files/medborgerpanel/files/nsd2401-ncp-documentation-report-wave-7.pdf

51.

Van der Veld

W. M.

Saris

W. E.

Satorra

(2009). Jrule 2.0: User manual (Unpublished manuscript, internal report). Radboud University Nijmegen, the Netherlands.

52.

van Meurs

Saris

W. E.

(1990). Memory effects in MTMM studies. In Saris

W. E.

van Meurs

(Eds.), Evaluation of measurement instruments by meta-analysis of multitrait–multimethod studies (pp. 134–146). Amsterdam, the Netherlands: North-Holland.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.46 MB