Fictitious Issues Revisited: Political Interest,Knowledge and the Generation of Nonattitudes

Fictitious Issues and Pseudo-Opinions

Perhaps the most vivid way in which the ephemeral nature of political attitudes has been demonstrated to date is through the elicitation of opinions on non-existent or highly obscure pieces of legislation. If we believe that people are willing to evaluate policy proposals without due cognisance of the relevant issues and facts, then we should expect to find significant numbers of respondents providing evaluations of policies with which they are, demonstrably, unfamiliar. While Howard Schuman and Stanley Presser (1980) found Stanley Payne's (1951) reference to 70 per cent of the public providing substantive responses to an item on a fictional ‘metallic metals act’ to be apocryphal, in their own original study around a third of respondents favoured or opposed the ‘monetary control’ and ‘agricultural trade’ acts. Although referring to genuine pieces of legislation at the time, these questions were considered so obscure as to be familiar to only a tiny fraction of the general population.

At the same time, an independent programme of research by George Bishop and colleagues at the University of Cincinnati found similarly high proportions of ‘pseudo-opinions’ ¹ in response to an imaginary ‘public affairs act’ (Bishop et al., 1980). Both sets of investigators also found that the proportion of respondents offering a substantive response to these items dropped from a third to around one in ten, when an explicit ‘no opinion’ filter was offered prior to administering the fictitious issue item. While some way from the majority that had been claimed by Payne (1951), then, a substantial proportion of the public were nevertheless willing to provide substantive responses to questions about non-existent legislation, even when given an explicit opportunity to report having no opinion at the outset. Subsequent replications in a range of different contexts confirmed the general robustness of these conclusions (Bishop et al., 1986; Graeff, 2002; Hartley, 1946; Hawkins and Coney, 1981; see also Hartley, 1946).

Explanations

What, then, leads people to report opinions on non-existent aspects of government policy? In addressing this question, most attention has focused on evaluating Philip Converse's memorable analogy of a mental ‘coin flip’ (Converse, 1964). Drawing on his analysis of opinion stability in the 1956–60 US national election panel study, Converse argued that ‘nonattitudes’, as he termed these responses, were generated much as one might stick a pin in a map. That is, he contended that large numbers of respondents choose a response alternative at random in order to conform to the protocols of the survey interview and to avoid appearing ignorant in front of the interviewer. The notion of random responding need not imply an equal probability of selecting each response alternative, as respondents may be drawn to particular response options through processes of acquiescence, satisficing and other response sets (Dillman, 2000; Krosnick et al., 2002). As Converse himself noted, a biased coin may yield more heads than tails in the long run but the generating mechanism is still essentially random (Converse, 1970).

Qualitative and statistical analyses of fictitious issues data, however, soon rendered the ‘mental coin flip’ account of pseudo-opinions highly implausible. Schuman and Presser, for instance, found that responses to their obscure items were correlated with confidence in the government and that spontaneous asides by respondents indicated that they were interpreting the question as if it related to a real issue. Similarly, Bishop and colleagues (1986) found the tendency to provide pseudo-opinions to be associated with the propensity to provide substantive answers on other items in the questionnaire and with existing partisan tendencies (Morin, 1995, cited in Bishop, 2005). Such findings clearly indicate that responses to un-cognised issues do not emerge unthinkingly, as if from a random number generator. Rather, ‘respondents make an educated (though wrong) guess as to what the obscure acts represent, then answer reasonably in their own terms about the constructed object’ (Schuman and Presser, 1980, p. 1223; see also 1980, p. 1223; Schwarz, 1995; Smith, 1984; Strack et al., 1991).

While this ‘imputed meaning’ hypothesis represents a strong refutation of the ‘mental coin flip’ account of nonattitude responding, it does not in our view problematise the main conclusion of the fictitious issues literature – that survey respondents are willing to provide opinions on issues about which they are not, indeed cannot, be aware. In doing so, respondents may well be anchoring the unfamiliar issue to an area on which they feel on firmer ground. However, this does not undermine the conclusion that some respondents are also doing this when surveyed on real issues of societal import. Having established that people will provide evaluations of policies that they have never heard about, the important question to be addressed next is: what are the social and individual characteristics that predispose individuals to do this?

Delineating the characteristics associated with pseudo-opinion responding more fully will enable us to develop a better understanding of the underlying response-generating mechanism. It is often assumed, for example, if only implicitly, that it is the less well informed and politically engaged who are most likely to provide ‘nonattitudinal’ responses. Lacking familiarity with matters politic, such respondents are motivated through pressures of social conformity to provide ‘random’ responses from the tops of their heads. As we have argued above, however, this ‘mental coin flip’ account no longer bears critical scrutiny. Instead, respondents who provide substantive answers to fictitious issues appear to be engaged in an active search for meaning within the context of current political debates and controversies, hardly the hallmark of the politically uninformed and disengaged (Delli Carpini and Keeter, 1996). Modelling the propensity to provide pseudo-opinions as a function of a set of covariates, selected on a theoretically informed basis, will enable us to elucidate the dispositional and situational characteristics that underlie these responses with greater clarity than has been evident to date.

The pattern and consistency of covariate relationships across different fictitious issue items is also of key interest in this regard. For it is not clear from existing research whether pseudo-opinion responding is a stable characteristic of individuals, or whether it is primarily a contextually dependent phenomenon. If the same sorts of individuals provide pseudo-opinions across a range of issue areas, then this would have implications for our understanding of the survey response process, as well as for questionnaire design and empirical analysis. If, however, the correlates of pseudo-opinion responding vary as a function of the topic area and/or other aspects of the research design, the implications are quite different. What, then, do existing studies tell us about the characteristics associated with pseudo-opinion responding?

In the Bishop et al. and Schuman and Presser studies, the two variables that emerged as the most consistent predictors of pseudo-opinions were ethnicity and education, with less educated and black respondents more likely to favour or oppose the non-existent legislation. In some conditions, men and younger respondents were more likely to provide pseudo-opinions, though these effects were rather weak and inconsistent across both issues and the filtered and non-filtered versions. For Bishop et al., the education effect is hypothesised to emerge because, ‘the less knowledgeable a person is about a given subject, the more easily he or she can be confused and pressured to give an opinion about it’ (Bishop et al., 1986, p. 248). By extension, of course, the more knowledgeable a person is about politics the more likely he or she will be to recognise, or suspect, that an issue is indeed fictitious. With regard to ethnicity, the mechanism is less clear, though Bishop et al. (1980, p. 204) speculate that it might arise due to black respondents wishing to ‘save face’ by not admitting ignorance to a white interviewer. ²

While Schuman and Presser found the same effects for education and ethnicity in their study, these were only significant in the ‘non-filtered’ versions. When respondents were given an explicit opportunity to say they had no opinion, the education and ethnicity differentials became non-significant. In their 1986 replication, Bishop and colleagues once more observed significant effects for education and ethnicity, though again only in the non-filtered condition. In the filtered condition, none of the respondent characteristics examined were found to predict the probability of pseudo-opinion reporting on any of the three issues. In sum, then, although it is now well established that people are willing to provide evaluations of obscure and non-existent political issues, we know rather little about the individual characteristics that predispose them to do so. The education and ethnicity differentials observed in the Bishop et al. and Schuman and Presser studies related only to the filtered versions of the questions. As opinion filters are rarely used in practice in contemporary opinion research, these findings must be considered of limited generality.

Our aim in this article, then, is to investigate the correlates of pseudo-opinion responding in more detail than has been attempted to date. In doing so, we seek to provide a clearer account of the social and individual characteristics that give rise to this kind of response – on both real and fictitious issues. Additionally, to our knowledge this article represents the first replication of the fictitious issues studies to be conducted on a general population sample outside the United States. The question of whether British citizens are willing to provide substantive responses to non-existent issues in similar numbers to their American counterparts is, therefore, of interest in and of itself. Once we have identified the sub-set of respondents willing to provide substantive responses to fictitious issues, the focus of the empirical part of the article moves to testing a set of theoretically derived hypotheses about the antecedents of pseudo-opinion responding.

In selecting variables for these predictive models, we include the basic demographic covariates that were examined by Schuman and Presser and Bishop and colleagues: age, sex and education, though we do not report the effects of ethnicity. This is because we are not able to obtain reliable estimates of the difference between black and white respondents, as our sample contains only 2 per cent of respondents who self-identified as black. In total 10 per cent of respondents identified themselves as being from an ethnic group other than ‘white British’ . However, these individuals are so ethnically heterogeneous that it makes no substantive sense to treat them as a single ‘non-white’ group (Nazroo, 2001). In any event, preliminary analyses showed there to be no statistical difference between the ‘white British’ and ‘other ethnic’ groups, so we do not include ethnicity as a predictor in the models to be presented later.

In addition to education, we also include a measure of political knowledge. During the period that the Bishop et al. and Schuman and Presser studies were conducted, education was generally used as a proxy for political sophistication. Since that time, however, education has been shown to act as a rather poor measure of the construct of interest (Converse, 2000; Zaller, 1991) and it is now conventional to use more direct measures of information holding, generally administered via a short factual knowledge quiz (Delli Carpini and Keeter, 1996). The expectation, however, is equivalent to that outlined for education: more politically knowledgeable respondents will be less likely to confuse the fictitious issue for a real one and, by the same token, will be more likely to suspect that it is not a real issue in the first place. Additionally, existing evidence suggests that more politically knowledgeable individuals are more reluctant to provide simplistic judgements on what appear to be complex policy issues (Krosnkick and Milburn, 1990). Our first hypothesis, therefore, is:

In addition to political knowledge, we also include a measure of self-reported interest in politics. Our rationale here is that some respondents may report opinions on non-existent issues because they feel that they are the sort of person that should have an opinion on political matters. It seems plausible to assume that it is those who identify themselves as being most interested in politics who are also most likely to consider that they should express an opinion, even on matters which might not seem immediately familiar. While this may appear somewhat contradictory with regard to H1, in that political knowledge is positively correlated with political interest, it is important to draw a distinction between political interest as a concept and its self-report by respondents in surveys. For there is strong evidence to suggest that self-reported political interest is highly susceptible to social desirability bias. Those who report being more interested in politics and election campaigns are significantly more likely to over-report on a range of what might be considered ‘civic’ behaviours. For example, Presser (1984) finds over-reporting across a number of different electoral behaviours, including voting, in the 1949 Denver Community Survey to be significantly higher among those who expressed more political interest (see also Cassel, 2003; McCutcheon et al., 2003; Vavreck, 2006). Our second hypothesis is then:

It is well known that self-reported interest in politics is highly sensitive to the context in which the question is asked. Using split ballot experiments, Bishop et al. (1984; 1982), for instance, show that reported interest in politics is significantly lower when the question is preceded by a series of difficult questions on factual political matters than when this order is reversed (see also Gaskell et al., 1993). It seems likely that this effect should generalise to questions eliciting opinions on government policy; respondents will be less likely to report interest in politics if they have just been asked their opinion on a political issue they have never heard of. Similarly, if respondents have reported a strong interest in politics prior to an obscure issue item, they will be more likely to attempt a substantive response, in order to maintain consistency with their previous answer. Our third hypothesis then becomes:

It is clear from previous research that, in making sense of unfamiliar issue items, a strategy employed by many respondents is to interpret the question through the filter of existing political preferences. Thus supporters of an incumbent government might choose to support a proposed piece of legislation on the grounds that it is likely to have been drafted by the party they support (Bishop, 2005). Similarly, respondents might interpret ‘the monetary control’ bill, for example, as relating to a tightening of fiscal policy, which might also result in responses to this item splitting along party lines. Thus we also include a measure of current vote intention as a covariate in the predictive model. While we suspect that this variable will influence the propensity to provide pseudo-opinion responses, we have no strong a priori hypotheses about how political affiliation will map on to support and opposition for the fictitious issues to be examined.

The final variable to be included in the predictive models is a measure of the respondent's level of self-confidence as a person. We include this variable because it seems likely that a tendency to provide opinions about issues one has never heard of is, to some degree, a function of personality. There are many potential aspects of personality that might ostensibly be linked with nonattitude responding. We have selected self-confidence on the grounds that this variable combines aspects of both self-esteem and attitude towards risk, both of which might be expected to correlate with pseudo-opinion reporting.

The expectation that pseudo-opinion responding will be related to self-confidence is also consistent with Bishop et al.'s (1980, p. 206) speculation that it takes ‘a fair amount of confidence in oneself to acknowledge that one does not have an opinion on something that sounds important’. In our view, however, the nature of the relationship is likely to be in the opposite direction to that suggested by Bishop et al. in that self-confident people are more likely to ‘chance their arm’ with an answer to a question, the meaning of which they are uncertain about. While there is also a potential inconsistency between this expectation and H1, in the sense that we might expect more politically knowledgeable individuals also to be more self-confident, the correlation between our knowledge measure and self-confidence suggests this is not the case. ³ Thus our final hypothesis is:

Data, Research Design and Ethical Considerations

The data for this study were collected as part of the Ipsos-MORI General Public Omnibus survey. This is a multi-stage Computer Assisted Personal Interview (CAPI) survey, covering a broad range of topics with a geographical coverage of mainland Britain. At the first stage, a regionally stratified sample of 210 parliamentary constituencies is randomly selected. At stage two, a government ward is randomly selected within each sampled constituency. Finally, ten respondents are selected purposively within each ward to match population marginals on age, sex, housing tenure and working status. The design is not, therefore, random but achieves a broad geographic coverage and matches the general population closely on a range of characteristics. Ipsos-MORI does not record refusals data, so it is not possible to report the AAPOR refusal rate. ⁴

Respondents were randomly assigned to one of four conditions. In conditions 1 and 2, they received the following question on the fictitious ‘Monetary Control Bill’ (MCB): Parliament is currently considering The Monetary Control Bill. Looking at this card, please tell me to what extent you support or oppose the passing of this bill?

This is a different response format from that employed in both the Schuman and Presser and Bishop et al. studies. In these studies, respondents were asked either to favour or oppose the bill, with no differentiation by degree and no middle alternative offered. For a random half of respondents, an opinion filter was administered prior to the fictitious issue items. We have not replicated these formats exactly here because neither format is, we believe, representative of the way that attitude questions are now generally asked in practice. Thus we have attempted to ask the fictitious issue questions in a format that might actually be used for this type of question on a real issue: as 5-point Likert items with a middle alternative and no explicit don't know option. Respondents spontaneously offering responses such as ‘I've never heard of it’ and ‘don't know’ were coded as such without probing.

In conditions 3 and 4, respondents were asked about the ‘Agricultural Trade Bill’ (ATB), using an identical question format. In conditions 1 and 3 a self-reported ‘interest in politics’ question was administered prior to the fictitious issues, while in conditions 2 and 4 this order was reversed. Prior to these two manipulated variables, all respondents were administered a set of standard demographic questions and asked which party they would vote for if an election were held the following day, and to assess their degree of self-confidence on a 5-point scale, running from ‘very confident’ to ‘very unconfident’ . Following the manipulated variables, all respondents were administered a three-item political knowledge quiz. Full question wordings are provided in the Appendix. We, therefore, have a 2*2 experimental design, manipulating question wording and order, as set out in Figure 1. These were the first items to be administered in the questionnaire, so we can discount the possibility that the observed distributions might have been affected by other item blocks in the questionnaire. ⁵

OPEN IN VIEWER

Figure 1:

Question Order across the Four Experimental Conditions

Our use of questions about fictitious pieces of legislation raises ethical issues relating to deception of respondents. In our judgement, the procedure we have employed is a rather modest departure from the standard practice of asking respondents about real issues, which they are known in advance to be broadly unfamiliar with. It is quite common, for instance, tosurvey the general public on their views about obscure areas of science and technology, which only a fraction of the public are likely to have heard about, let alone developed an opinion toward (Gaskell et al., 2004). Of course, such studies are different from the present one in the important respect that the unfamiliar issues asked about are real. Nevertheless, their widespread use demonstrates that, in general survey practice, it is not thought unethical to ask the public about issues that many or most individuals will be completely ignorant of.

The tacit acceptance that it is unproblematic to survey the public on obscure issues opens up an alternative strategy to the study of ‘pseudo-opinions’ . That is, to ask respondents about issues which are real but so obscure that it can safely be assumed that the vast majority of the public will not be familiar with them. Indeed, this is the strategy adopted by Schuman and Presser (1981) in the study which first used the (then) real but highly esoteric ‘agricultural trade’ and ‘monetary control’ acts. On closer inspection, however, the practical difference – in terms of the respondent experience – between the Schuman and Presser approach and that adopted by Bishop and colleagues, in which fictitious legislation is used, is difficult to discern. In practice, both approaches survey the public about issues they are extremely unlikely to have heard about, in order to estimate the likely prevalence of ‘nonattitudes’ on genuine and salient areas of public debate.

In this sense, both approaches ‘deceive’ respondents about the true purpose of asking the issue questions and both approaches might result in a degree of confusion among similar numbers of respondents, by asking them about issues they have not have heard of. In scientific terms, however, the Bishop et al. procedure is clearly preferable. This is because, under the Schuman and Presser approach, one must make the assumption that respondents who provide substantive answers have not heard of the issue in question, while under the Bishop et al. procedure, this is known in advance. Thus, the practical difference – in terms of ethical treatment of respondents – between our procedure and one that is generally considered unproblematic is small, yet the scientific case for preferring it is strong. In sum then, although one should never adopt a research strategy which involves deception of subjects lightly, we believe that if the potential for harm is low and the scientific case strong, it is possible to conduct such research in a manner that is justifiable by consensually accepted norms of practice (Gorard, 2002).

In order to mitigate any potential confusion or distress among respondents who might believe or suspect that the issue they are asked about is fictitious, we implemented the following procedure. For all respondents who answered ‘don't know’, or indicated that they had not heard of the issue, the interviewers read out the following script: ‘You may have noticed the question about the agricultural trade/monetary control bill refers to a piece of legislation which does not actually exist. This question was included as part of an ongoing programme of methodological research by the University of Surrey’ . Respondents were also offered the opportunity of contacting the research team at the University of Surrey for further information, although none took up this option.

This script was not read to respondents who provided substantive responses to the fictitious issue questions, because it seemed likely that revealing the fact that he or she had provided an opinion on a non-existent piece of legislation would be embarrassing for the respondent. Thus, revealing the nature of the deception after the event would, we felt, be potentially more harmful to such respondents than leaving them believing that they had been surveyed about an issue which they (incorrectly) interpreted as being real.

Results

Table 1 shows the marginal distribution for both items. For the MCB item, the proportion of substantive responses is 15 per cent, while for ATB it is 11 per cent. These figures are considerably lower than those obtained for the unfiltered fictitious issue items by both Bishop et al. (1980; 1986) and Schuman and Presser (1980), where around a third of respondents provided substantive responses.

There are, of course, many reasons why this difference might have arisen. One is cultural; Americans may simply be more willing to offer opinions on things they know little about. While this interpretation might fit with the prejudices of certain sections of the British media, we suspect that a more likely source is our use of a middle alternative, which attracts between a fifth and a quarter of respondents on the two items, respectively. It would seem that, in the absence of an explicit opinion filter, many respondents select the middle alternative as a way of registering uncertainty without having spontaneously to admit ignorance. Thus the middle alternative appears to act as a weak opinion filter, making our results more comparable to the ‘filtered’ versions of these questions reported in the earlier studies. There is also a mode difference between this and the earlier studies and it is certainly possible that people may be more willing to provide pseudo-opinions when not physically co-present with the interviewer. Our data do not, however, afford any leverage on whether mode has an independent effect in this regard.

On the ATB item, ‘opinion’ is evenly split, with almost exactly even numbers supporting and opposing the fictional bill, while on the MCB the majority substantive position is in support of the bill, by a ratio of 2 : 1. As in previous investigations, this demonstrates that responses to fictitious issue questions are not random, in the sense of having equal probabilities across the available answer categories. On the MCB there appears to be some aspect of the wording or context in which the question is asked that is leading people to favour rather than oppose the bill. Whatever this feature might be, it would appear not to be universal in nature, as these distributions are the opposite of those reported by Schuman and Presser (1980), who found a majority in favour of the ATB, and even numbers favouring and opposing the MCB.

OPEN IN VIEWER

Table 1:

Marginal Distributions for Monetary Control and Agricultural Trade Items

	Monetary control (%)	Agricultural trade (%)
Strongly support	0.8	0.4
Tend to support	9.7	5.2
Neither support nor oppose	26.1	20.7
Tend to oppose	3.6	4.1
Strongly oppose	1.2	1.3
Never heard of it	45.6	55.6
Don't know	13.0	12.8
(n=)	1,017	1,024

Are respondents interpreting these non-existent pieces of legislation in ways that enable them to use existing political preferences to come to an evaluative judgement? We can get some handle on this by examining the distribution of substantive responses against reported vote intention (Figure 2). For the MCB item, Labour supporters are less likely to support the bill than are supporters of the other two main parties. Although this difference is not statistically significant, it is suggestive of the idea that a ‘monetary control’ bill may imply to some respondents a proposed tightening of fiscal policy, which would generally be less favoured by those on the left of British politics.

OPEN IN VIEWER

Figure 2:

Substantive Responses to Fictitious Issue Items by Vote Intention

For the ATB item, a near opposite pattern emerges, with Labour supporters significantly more likely to support the named bill. It is likely, in our view, that this difference emerges because respondents interpret the ATB item as relating to Third World aid and debt, an issue which was high on the political agenda at the time this survey was conducted. The fact that those on the left of British politics generally advocate higher levels of aid and the reduction of debt fits with this interpretation. While these are speculative explanations of the patterning of party support across these items, our results nonetheless support the conclusion of existing studies that some respondents attempt to make sense of fictitious and obscure items by setting them within the context of current political debates. This enables them to come to a view on the basis of existing party loyalty.

Next, we turn to a consideration of the characteristics of respondents which predispose them to providing substantive responses to these items. Table 2 shows for each issue a binary logistic regression of pseudo-opinion response (where any substantive response is coded ‘1’ and all other responses are coded ‘0’) on the covariates described earlier. The first thing to note about Table 2 is the substantial difference in the pattern of significant predictors; while only two independent variables are significant for the MCB item, there are six significant effects for ATB. Clearly, then, the factors underpinning pseudo-opinion responses are not fixed, but vary as a function of the apparent meaning of the fictional piece of legislation or issue.

OPEN IN VIEWER

Table 2:

Binary Logit Models Predicting Probability of Pseudo-Opinion Reporting

	Monetary control		Agricultural trade
	(n = 1,017)		(n = 1,024)
	Logit (se)	OR	Logit (se)	OR
Political interest	0.32 (0.14)*	1.38	0.98 (0.20)***	2.66
Order (1 = issue first)	0.19 (0.55)	1.21	1.42 (0.73)	4.12
Political interest*order	−0.09 (0.20)	0.92	−0.56 (0.25)*	0.57
Sex (1 = male)	0.10 (0.18)	1.10	0.43 (0.22)*	1.54
Age (in years)	−0.01 (0.02)	0.99	0.06 (0.03)	1.06
Age squared	0.00 (0.00)	1.00	0.00 (0.00)	1.00
Highest qualification (1 = degree)	−0.09 (0.24)	0.91	0.02 (0.26)	1.02
Vote intention
Labour	0.19 (0.23)	1.21	0.60 (0.27)*	1.82
Liberal Democrats	0.25 (0.28)	1.29	−0.02 (0.37)	0.98
Conservatives	0.15 (0.24)	1.17	0.82 (0.27)**	2.28
Self-confidence	0.33 (0.13)*	1.39	0.02 (0.14)	1.02
Political knowledge	−0.17 (0.10)	0.84	−0.25 (0.12)*	0.78
Constant	−1.86 (0.71)**	0.16	−6.35 (0.96)***	0.00

***

p < 0.001;

**

p < 0.01;

*

p < 0.05.

Source: Ipsos-MORI Omnibus Survey August 2006.

In terms of our first hypothesis, H1, the coefficients for political knowledge are both negative, as predicted, though only for the ATB item is the coefficient significantly different from zero at the 95 per cent level of confidence (for the MCB item, the coefficient is significant at p < 0.1). Thus we have partial confirmation of this hypothesis. The marginal non-significance of political knowledge on the MCB item may reflect the rather unsatisfactory nature of our political knowledge measure. Limited questionnaire space restricted us to a knowledge scale comprising only three questions, and with a larger pool of items it is likely that the effects would be larger and more robust.

Our second hypothesis, H2, is also confirmed; the partial correlation between political interest and the probability of pseudo-opinion responding is positive and significant for both items. For the ATB item, the odds ratio for the main effect shows this to be a particularly powerful effect, with the odds of providing a substantive response increasing by around 150 per cent for each unit increase in political interest. For both items an interaction in the expected direction is observed, though this is only significant for the ATB item.

Thus, our third hypothesis, H3, is also partially supported – the effect of self-reported political interest on pseudo-opinion responding is greater when the interest item is placed before the fictitious issue question. This interaction can be seen more clearly in Figure 3, which plots for each item the predicted probability of a substantive response as a function of political interest and response order.

OPEN IN VIEWER

Figure 3:

Predicted Probability of Pseudo-Opinion by Political Interest and Response Order

Our last hypothesis, H4, is also partially confirmed, although this time the effect is only significant for the MCB item. Respondents who report higher levels of self-confidence are more likely to provide substantive responses on the MCB. For the ATB item, however, the coefficient is weak and a long way from reaching statistical significance.

In terms of consistency with the findings of previous research, we too find pseudo-opinion responses to be more prevalent among men, though this difference is only significant for the ATB item. We find no effect of age or education for either item. Finally, pseudo-opinions on the ATB are significantly more likely among supporters of either of the two main parties, relative to non-voters, voters for minority parties and those who have not made up their minds. Thus party support can influence the probability of providing a substantive response at all, in addition to influencing the direction of the expressed opinion. Again, however, this effect is contingent on the nature of the particular issue in question, as no party support effect is apparent for the MCB item.

Discussion

Most responses to Schuman and Presser and Bishop et al.'s troubling observations that many ordinary citizens are willing to provide substantive responses to non-existent issues have focused on rejecting the notion that the generating mechanism is random and unthinking.

It now seems clear that responses to fictitious issue questions are not, for most respondents, equivalent to mentally flipping a coin. Instead, respondents who provide substantive answers to these questions appear to be involved in an active search for meaning, both in the wording of the question and in the overall context in which it is asked. Once they have formed a judgement on what they think the question is about, they fall back on existing attitudes and partisan tendencies to arrive at a substantive evaluation.

The results we have presented here lend further support to this ‘imputed meaning’ account of the social-cognitive basis of pseudo-opinion responses. We found that the division of ‘opinion’ on the two fictitious issues examined split along party lines, in a way that indicates that respondents were anchoring the fictional bills to genuine issues of public controversy. Additionally, our study demonstrates that substantive responding to non-existent issues is not a uniquely American phenomenon; between 10 and 15 per cent of the British public offered opinions on the non-existent monetary control and agricultural trade acts. These figures are approximately equivalent to those found in the American population, when a ‘filtered’ version of the questions was employed.

However, dismissing the ‘mental coin flip’ account of fictitious issue responding should not lead us also to reject the core assumption of these studies; that asking questions about non-existent issues accurately reproduces the conditions in which respondents are surveyed on real issues, of which they are largely or wholly unaware. Rather, our results should be taken as robust evidence that questions addressing diverse areas of government policy are likely to be ‘contaminated’ by responses that are based on partially or wholly irrelevant considerations, as many have long suspected. Having established the robustness of this conclusion, the key question we have sought to address in this article is: what type of person is most likely to provide substantive responses rather than admit ignorance when faced with a question on an ambiguous or unfamiliar topic?

In addressing this question, our findings diverge to some degree from those of existing studies. We found no difference in the propensity to provide substantive responses to the fictitious items by ethnicity, education or age. Part of the reason that ethnicity had no effect in our study is a function of the different ethnic compositions of America in the 1980s and Britain in 2006. These populations are simply not ethnically equivalent, so our findings have little bearing on the validity of these earlier estimates. And although we observed no effect of education, we did find that more politically knowledgeable respondents were less likely to provide a substantive response to the fictitious issue questions. As the Schuman and Presser and Bishop et al. studies were implicitly using education as a proxy for political knowledge, our findings are perhaps more convergent than might initially seem the case.

In addition to political knowledge, another variable tested for the first time as a predictor of nonattitude responding in this study was personal self-confidence. Although the hypothesis that more self-confident individuals would be more likely to provide opinions on these issues was only supported for the MCB item, we interpret this as generally supporting the view that personality differences are part of the causal mechanism. Our findings are suggestive of the idea that those higher in self-esteem and more willing to take risks are also those more likely to believe that they have correctly identified the meaning of an ambiguous question.

The most substantial effect we observed, however, was for self-reported interest in politics, with higher levels of political interest associated with a considerably greater probability of a pseudo-opinion response. This result is consistent with research showing that self-reported political interest is associated with over-reporting across a range of electoral behaviours (Cassel, 2003; McCutcheon et al., 2003; Presser, 1984; Vavreck, 2006). These findings lend further support to the suspicion that self-reported political interest is not a ‘pure’ measure of an individual's motivation to acquire and retain information about politics. Instead, it appears to incorporate a significant element of socially desirable responding for those individuals and groups which hold civic attitudes and behaviour in high esteem.

Additionally, we found the effect of self-reported political interest to be accentuated when it was administered prior to the fictitious issue item, rather than when this order was reversed. Together, these results suggest that observed associations between expressed interest in politics and policy attitudes are likely to be artificially inflated in many, perhaps most, instances. More worryingly, perhaps, our findings imply that if political interest questions are placed towards the beginning of a questionnaire, as is common in many political attitude surveys, this may serve to inflate the proportion of nonattitude responses to policy opinion questions administered later in the questionnaire. Another implication of our findings for the design of political attitude surveys is that, for remote and non-salient areas of public policy, it would seem sensible to consider employing some form of opinion filtering – a practice that is more often referred to than implemented in contemporary questionnaire design. The reason for the low uptake of opinion filtering approaches is primarily related to the higher rates of missing data yielded by this approach. While missing data are no doubt undesirable from the perspective of the data analyst, our findings suggest that if we are merely replacing ‘no opinion’ responses with pseudo-opinions, the envisaged gain may well be illusory.

The fact that a more robust pattern of significant findings was observed for the ATB than for the MCB item might be taken as contradictory to the imputed meaning hypothesis. We would be inclined to resist such an interpretation. For the differential pattern of coefficients across the two items is equally consistent with the view that there is simply greater heterogeneity in respondent interpretation of what the MCB item actually relates to. If respondents are indeed imputing more variable meanings to the MCB item, then we would expect to observe weaker and more variable covariate relationships, relative to the more homogeneously interpreted ATB item. Because we were limited to examining just two items in the current study, both these accounts are in the realm of speculation. By the same token, however, this indeterminacy suggests that a useful strategy for advancing understanding in this area would be to manipulate the consistency with which respondents are likely to interpret fictitious issues, possibly via randomising the content of preceding items across experimental conditions. Retrospective probes eliciting verbatim accounts of respondent interpretations would also be likely to shed useful light on this question. Lastly, it goes without saying that future investigations could usefully expand the range of variables considered as predictors of pseudo-opinion responding, for those we have been able to consider here clearly do not represent an exhaustive list.