Gaining insights without questions;Political Polling;Democracy Awards and IJMR Executive Editorial Board

A summary of the findings

Patrick Sturgis (University of Southampton/NCRM) opened the seminar by describing the background to the 2015 British Polling Council (BPC)/MRS Polling inquiry, and the subsequent Lords Select Committee on Political Polling and Digital Media investigation.

Sturgis summarized the recommendations from the Lords Select Committee, identifying those specifically for the BPC:

And those recommendations for others:

Lord Lipsey, who chaired the Lord’s Select Committee which reviews polling and digital media, responded to Sturgis’ opening summary and reflected on the Lord’s deliberations on the merits of statutory regulation for polling, and that such a recommendation would have been a “waste of time” as UK Ministers made clear this was not an outcome they would support.

Lipsey noted the 2015 BPC/MRS report was very valuable, but did result in some of the problems of the 2017 election due to the adjustments made as a result of this report. Lipsey also noted the impact of response rates and turnout, and expressed some dissatisfaction on the limitations of margins of error when non-random sampling is used. Lipsey reiterated that it was the “Last Chance Saloon” for pollsters to get their house in order; if they don’t, there could be more serious problems ahead.

The response from the BPC

Sir John Curtice, as President of the BPC, took to the floor to discuss the implications of the report for BPC stressing the self-regulatory status of the organization, its limited financial resources, and that its activities are undertaken on a voluntary basis. Curtice stressed that BPC does not adjudicate on the merits of sampling methodology or questionnaire design, nor does BPC provide formal advice on how polls should be conducted or evaluate quality of media reporting. Curtice reflected on the Lord’s recommendations and identified those within and those beyond BPC’s purview. The latter activities raise questions about BPC’s remit: should BPC be about being an information provider/educator or the public and media?; plus take responsibility for the quality of polls, conduct of polls, policing the conduct of individual polls, facilitating the reporting of polls, and policing the reporting of polls?; and if so what should be its structure and resourcing?

MRS view

Jane Frost, the MRS CEO, responded on behalf of MRS. Frost reflected on past-Her Majesty’s Revenue and Customs (HMRC) research and the numeracy levels of the British public, which is estimated at around the equivalent of 7 years old and how this impacts people’s ability to interpret and understand election polls. Frost also highlighted the coverage of the MRS Code of Conduct, particularly rule 33, which sets the ethical rules for data collection design which is supported by MRS training and qualifications. Frost also referred to the MRS supplementary guidance on polling to assist non-practitioners in understanding how to read opinion polls plus MRS’ long-standing advisory and complaints and disciplinary approaches.

A journalist’s perspective

John Rintoul, Chief Political Commentator for The Independent, gave a journalistic view. Rintoul stressed that banning pre-election poll is one of the “stupidest ideas” based on a misunderstanding of freedom of expression. Journalist’s experience of the Levinson media inquiry indicates how the media would react to the suggestion of BPC being involved in the reporting within the media of polls! Generally, media has been reasonably responsible. The other half of the Lord’s review—the behavior of social and the digital media—is a different issue. Some social media uses opinion polls to influence public opinion, but that is marketing not research, and is why transparency is essential. Rintoul also reflected on the challenges of educating the reader, distinguishing between self-selected polls and representative polls, and noting some gray areas such as the political party polls which are restricted to members of the political party.

Practitioners’ response

Thereafter, some research and polling practitioners responded. First up Ben Page, Chief Executive of Ipsos MORI, reflected that trust in pollsters remains well above government ministers, bankers, and journalists. Research and polling is more than just about voting intentions. Research is about other activities such as observation, and so on. Context and information about policy issues is important when assessing question wordings, bias. Page gave examples of confirmation bias, while acknowledged the need to continue working on accuracy; in effect, pollsters have to “continue fighting the last war” and reflected that with the benefit of hindsight, pollsters should have said the results were too close to call rather than declaring a result.

The final speaker was Joe Twyman, formerly of YouGov, now one of the Founder’s of Deltapoll. Twyman reflected that six electoral events had occurred in the last 1,476 days before this polling event. Twyman noted that YouGov were the first pollsters to predict a hung parliament, which met with some derisory social media comments about the organization. It was also noted that private pollsters can make claims that their polls accurately predicted results as they are not published, so any assertions cannot be refuted.

Still work to do

Lord Lipsey closed proceedings with some final observations, noting that the recommendations are not the most important part of the Lord’s report rather that the report sets out the state of the polling industry. Lipsey stressed that the Lords are not recommending anyone “polices” the polling industry rather that there is better coordination of the setting of rules. Lipsey also reflected that in his view there was too much pessimism about media’s willingness to engage with this topic; IPSO and IMPRESS are sympathetic of the need for journalistic training and recognize the need for the media to adhere to the disclosure rules. Lipsey concluded he was “mildly Whiggish” about media improvements. You will also find a paper on polling predictions in this issue by Johnson et al.

(Reporting for IJMR by Debrah Harding, Managing Director, MRS)

IJMR Collaborative Award

The IJMR Executive Editorial Board selected the paper “Alternative methods for selecting web survey samples” (published in IJMR 60/4) by Melanie Revilla (Research Center for Survey Methodology, Universitat Pompeu Fabra, Barcelona) and Carlos Ochoa (NetQuest, Barcelona, Spain) for the 2018 IJMR Collaborative Award. In selecting this paper, the Board felt that

the authors explored an important topic—whether collecting additional profile data, or passive data of internet usage collected via meters, could improve the efficiency and data quality of internet panel-based surveys. This could reduce the number of panel members being rejected in screening questions and help address the potentially negative impact of this on attitudes towards being a panel member. From the comprehensive literature review, the authors developed hypotheses tested through a rigorous research design, leading to clearly described and actionable findings.

MRS Gold Medal

I’m also very pleased to announce that Bill Blyth was awarded the coveted MRS Gold Medal for over 45 years of outstanding contribution to the social and market research sector (https://www.mrs.org.uk/article/mrs/mrs-gold-medal-awarded-to-bill-blyth). Bill has had a long association with IJMR until his retirement from TNS. We have published papers authored, or co-authored, by Bill and for many years he was a member of the Editorial Advisory Board. Two papers by Bill were published in the special issues of Journal of the Market Research Society (JMRS) (Vol. 38/4 and Vol. 39/1) commemorating MRS 50th anniversary, one of which “How do you like your data: raw, al dente or stewed,” co-authored with Tim Bowles, was re-published online as an IJMR Landmark paper (https://www.mrs.org.uk/blog/ijmr/landmark-paper-how-do-you-like-your-data-raw-al-dente-or-stewed), and two of his contributions won the MRS Innovation in Research Methodology awards.

MRS Silver Medal

Finally, the MRS Silver Medal. This is an annual award for the best paper published in IJMR and was awarded to the paper by Geoffrey Roughton and Iain MacKay, “Are interviews costing £0.08 a waste of money? Reviewing Google Surveys for ‘wisdom of crowds’ projects,” published in IJMR 59/2 (March 2017). This was published in our Special Issue, containing five papers based on ones presented at the ASC International Conference held at the University of Winchester in September 2016. In selecting this paper, the judges commented,

this iconoclastic and useful paper provides robust, methodologically sound, actionable insights about the use of low-cost Google surveys—working in conjunction with “wisdom of the crowd” theory—that give us valid and reliable guidance on how to use this technique to explore likely future events. Written in an authoritative and accessible way, it will make a valuable contribution to the honing of our sector’s skills. More than this, the judges felt that in publishing something quite contrary to the accepted principles of research, the authors were daring to do this work in the first place and their honesty in sharing their failures when they did occur was brave.

Challenges in predicting election outcomes at constituency level

Our Special Issue, 60/2 (March 2018), focused on the “Challenges of accurately measuring public opinion.” In 60/3, we subsequently published a Conference Note on the same theme, and our first paper in this issue also returns to this theme. Johnson (University of Bristol); Rossiter (independent); Pattie, Todd (University of Sheffield); and Manley and Jones (University of Bristol) discuss the potential for applying constituency-level predictions of voting intentions in addressing the challenge of forecasting the number of seats parties will achieve, rather than the national vote share outputs, from opinion polls. The authors discuss the growing importance of marginal constituencies in deciding the outcome of a UK general election, the attention that political parties pay to them, and the range of factors that can influence the outcome within any particular constituency. They also discuss the impact of polls undertaken in key marginals, such as conducted by Lord Ashcroft on behalf of the Conservative Party, on strategies and outcomes. The authors also describe the trend toward Internet panel-based non-probability-based polling and the resultant meaningless use of confidence intervals as a measure of accuracy (see Peterson, “On the myth of reported precision in public opinion polls,” IJMR 60/2) that are also reflected in reports on opinion polling methods from the AAPOR in the United States and the BPC/MRS in the United Kingdom. The authors argue that Multilevel Regression Post-stratification methods (nicknamed “Mister P”) based on a combination of the findings from polls conducted in the 2015 election campaign plus local data from other sources could have produced a more accurate forecast in 2017 for most constituencies. They discuss the three sets of estimates using this method for marginals created by YouGov (68 correct predictions in England and Wales), Lord Ashcroft (53), and Chris Hanretty (42), discussing the analysis for estimates in England/Wales and Scotland. The results show mixed success, and the authors discuss how the outputs might be refined. Finally, the authors conclude that constituency-level predictions will become the focus of activity, by pollsters and political parties, with marginals becoming center-stage, leaving other electors disregarded.

Increasing efficiency in online sampling

As Gómez and Porcar (NetQuest, Barcelona, Spain) discuss in their paper, increased competition in the online survey market is causing researchers to look at ways to reduce survey costs and improve efficiency. Response rates continue to fall, panels suffer from attrition and the impact of discarding participants, recruiting costs are rising, and the costs of providing incentives also need to be closely managed. The authors’ objective is to help researchers identify the issues and develop solutions, their three aims being, first, to understand the effect of using quota sampling on fieldwork efficiency; second, to measure the effect for different quota settings; and third, to design new strategies to improve fieldwork efficiency. They begin with a discussion on sampling methods, and the challenge posed by asynchronously interviewing in online panels quota samples, with discard levels varying between 5% and 80%, compared to fieldwork surveys. The main methods used in online quota sampling are described (interlocking and non-interlocking). In further sections, the authors describe and discuss an analytical model to estimate fieldwork efficiency, the impact on efficiency of different quota settings, and strategies to improve efficiencies. They conclude with three recommendations for further research in this field.

Measuring preferences and values in mail surveys

While the Internet has become a, if not “the,” primary data collection mode in many countries (44% overall in ESOMAR 2017 Global Market Research Report, 2017), mail-based surveys are still commissioned, even though the volume of research using this method has fallen to 1% internationally (ESOMAR). The authors focus on the challenges of using ranking-based questions, often used to identify preference and values in mail surveys. Smyth, Olson (University of Nebraska–Lincoln, USA), and Burke (LPC Consulting Associates, Inc., USA) discuss the differences between ranking and rating question formats, and the difficulties in developing rating formats in mail, compared to online surveys, leading to incorrect completion, satisficing, and skipping. A split-sample experiment is described testing “write-in numbering” against “most-second most” formats. The experiment was conducted using the National Health, Wellbeing and Perspectives Survey, with 6,000 participants split into two treatments. Also tested were the timing of incentives and the method for selecting the participant within household. The authors found that both methods led to high levels of response error, affecting the findings, especially where the “most-second most” format was used, participants more likely to incorrectly apply their experience from taking part in other surveys. They caution against using ranking format questions in mail surveys.

Developing effective screening to identify members of consumer collectives

Hawkins (ICN Business School, Nancy, France) focuses on the challenges in identifying members of consumption collectives, and the subtle, but important, differences in status that need to be recognized in research projects, and marketing communications. The author defines members as “a group of consumers who share a commitment to a product class, brand, activity or consumption ideology.” As the author also describes, this covers a wide spectrum of association, including peripheral participants who aim to, over time, become legitimate members. Screening for participation is therefore important, if true members are to be identified from those with lesser, or different, affiliations. The author argues that the need for such screening is often missing from studies in this field. The author conducted his qualitative research at the annual Bikes Blues and BBQ (BBB) festival held in Fayetteville, Arkansas, in the United States (https://en.wikipedia.org/wiki/Bikes_Blues_and_BBQ), which attracts over 250,000 visitors with a wide variety of interests and levels of affiliation and discusses why this event provides a suitable location for research in this field. At the core of the paper is a discussion of the main categories of consumption collective (brand community, consumption tribe, communities of practice), illustrating the differences using the research findings. The paper also includes a screening questionnaire, demonstrating how it helps in differentiating between different levels of participation.

The relationship between power state and status consumption

In the final paper, Jin, Xu, and Wang (Jilin University, Changchun, China) examine the impact of power status on status-related consumption. They define the topic as the “acquisition, display, or use of products that are implicitly or explicitly associated with a position in the social hierarchy.” In particular, the authors focus on the status of the individual and how it relates to the status of others when making consumption choices: are low-power state consumers more inclined to participate in status consumption than high-power state consumers when the status of others is superior, and are high-power state consumers more inclined to participate in status consumption than lower state consumers when the status of others is inferior. Two experiments are described: one to test out the effect of manipulating the status of others and the other related to the likely purchase of a particular luxury product. The findings demonstrate that power state and social comparison interact to influence consumer choice, including a reaction to the status of others. The authors conclude with a discussion of the implications for marketing strategy in differentiating communications to consumers with different status and different status needs in relation to others.

Identifying conversation patterns in social media

A version of our Forum article won the Best Methodological Paper award at the ESOMAR Congress in 2015. In the paper, Findlay (TNS, South Africa) uses Twitter to describe and discuss the morphological structure of Twitter network topologies, the patterns of social conversations on Twitter, at brand, campaign and category levels, identifying the new tools that can be used in social media analysis to evaluate campaign performances and brand image. The author describes the theoretical foundations, the factors contributing to content being shared, and the inherent difficulties in predicting what content might go viral. Network typologies are described along a spectrum and the six types of conversation identified in research by the Social Media Research Foundation. At the core of the paper are detailed descriptions of the interaction networks (footprints) at brand, campaign, and category levels that can be identified in Twitter conversations, across a spectrum from where the brand controls the content to situations (highly centralized) where the control is in the hands of consumers (highly decentralized). With increased experience, strategies mature providing real engagement with consumers, encouraging greater interaction between users. The author acknowledges that the paper primarily provides a qualitative perspective and the next step will be to identify a method to quantify these conversational types.