Abstract
In this editorial, I analyze the state of the market research sector and consider future directions. We are seeing a growth in evidence-based decision making and a shift toward customer centricity. This is leading to more surveys, more depth interviews, more focus groups, more ethnography, more usability testing, more semiotics, more build-test-learn, and more analytics. Indeed, more of almost anything that allows organizations to leverage insights about what people want, experience, and believe. However, less and less of this research is being done by people who call themselves “market researchers” doing tasks that they describe as “market research.”
Follow the money
Following the money is a good way to get an insight into what is happening to the research world. Drawing on the 2020 ESOMAR Global Market Research Report, we see that there are two dramatically different pictures of the insight’s world. Looking at ESOMAR’s traditional picture (focusing on things like surveys and focus groups), we see a global industry of about US$46 billion. However, the report also includes an alternative picture that includes tech-enabled methods of achieving insights, which takes the total for the insight ecosystem to about US$90 billion. This growth in the tech-enabled part of the research world has been noticeable for the last few years and we have now passed the tipping point. More than 50% of the insight’s ecosystem is not traditional market research.
Another key data point is revealed by the ESOMAR and partners’ User & Buyers Global Insights Study 2020 & 2021. This shows a shift from research being conducted externally (e.g., via agencies) toward it being done internally. The data indicate that about 50% of research projects are being conducted internally and the trend is for a larger share to move internal. Note, there are exceptions, with some organizations moving the other way, but the trend is large and growing.
From whale to minnow
What the money trail reveals, looking back over 20–30 years, is that market research has gone from being a whale in the past to a minnow today. The absolute size of the market research industry has grown, the largest agencies have grown, but our share of the insights ecosystem has become smaller and smaller. Go back 30 years; most surveys were produced by market research agencies; most focus groups were moderated by qualitative market researchers. Today, most surveys, most focus groups, and most online discussions are not being created and run by people who call themselves “market researchers.”
How?
I will discuss why the changes have happened later, but I think we need to understand the how first. The simple answer to the “how?” question is online platforms—often referred to as SaaS (Software as a Service).
As an example of the change, consider surveys in 1991 and 2021. In 1991, the use of surveys required one or more of: (1) the ability to produce paper questionnaires, to distribute them, to get interviewers to administer them, to punch the results, and write a tabulation specification; (2) for telephone you needed a CATI (Computer Aided Telephone Interviews) system and call center—with all the associated scripting and analysis skills; and (3) for CAPI (Computer Aided Personal Interviews) you needed people moving computers around the country to collect data, scripters, interviewers, logistical support, and all the analysis processes.
Now jump to 2021, and the picture has changed completely:
The sample is no longer “fresh,” it tends to be from panels, or customer lists, or communities.
Tools like Survey Monkey, QuestionPro, Prolific, and Qualtrics are easy to use and powerful. Many of these platforms also have the option to add a sample from the same supplier (or via a marketplace, operating behind the scenes).
You can bypass the sample and survey process entirely by going to a pre-built solution (such as Zappi). For example, you can run a concept or ad test where all you do is pick a market, upload materials, and receive back an automated report.
The frog in the well can’t understand the Qualtrics valuation
The growth in what has been labeled “ResTech” has led to some very different valuations for companies operating in the research world. When Qualtrics was purchased by SAP for US$8 billion, most market researchers scratched their head and were convinced that it could not be worth so much. Now Qualtrics is going for a new IPO and is currently projected to be worth US$15 billion—remember that the entire traditional, global market research industry has a turnover of only US$46 billion. The contrast is even more marked if we compare research giant Ipsos with CX specialist Medallia. Ipsos has a turnover of about US$2.5 billion and has a market capitalization of about US$1.5. Medallia has a turnover of about US$0.5 billion but a market capitalization of US$4.5 billion.
The reason this valuations seem so high to market researchers is akin to the Chinese story of the frog in the well (井底之蛙). The frog lives at the bottom of a deep well and assumes the light at the top of well is the sun. One day a bird flies down into the well and tells the frog that there is a whole world outside, but the frog disbelieves the bird and stays in the hole.
The valuations of companies like Qualtrics, Survey Monkey, Medallia, and others seem high if you think they are in the “market research” business. If you see them as being in the customer centricity, customer-connected, customer-obsessed business, then the valuations make much more sense. The key to these platforms is not that they empower researchers to do their job better. The key to these platforms is that they empower non-researchers to do research-related tasks.
Why?
Having tackled the how, it is time to consider the why. Why is research becoming more democratized, why is it becoming more internal, and why is it becoming more automated? There are two drivers that have caused the change to take place: (1) customer centricity and (2) agile.
Organizations have embraced customer centricity as a core belief. The most common refrain we hear today is all about “putting customers at the heart of everything.” When an organization adopts customer centricity, everybody wants/needs to connect with customers. In the past, connecting with customers was the job of the insights team, but with customer centricity everybody needs to connect with customers, for example, operations, HR, sales, communications, logistics, and design.
Agile has many meanings, but all of them convey a need to work faster, typically using straightforward methods, utilizing efficiency rather than depth, in cost-effective ways. Teams are applying design thinking, or lean start-up, or iterative methods of focusing on customers. A process that is described by Jeff Gothelf in “Lean vs Agile vs Design Thinking” as “less research, more often.”
Democratization
One word that describes the direction of travel is democratization, a process whereby the creation and use of insights spreads out from a center of knowledge to the rest of the organization. The positive side of this change is that evidence-based decision making becomes embedded throughout the organization, without building barriers of process, time, and budget.
However, the challenge is that the creation and usage of insights have moved, and is moving, into the hands of people who (1) have no specific training in how to create and use insights and (2) who often have no direct access to people who can provide support and advice.
The way that insights are being democratized rests heavily on the use of platforms, with standardized projects, and automated reports. This process favors well-established, straightforward approaches and shifts the balance away from the innovative, bespoke, specialist approaches (such as neuroscience and behavioral economics) toward short, direct surveys and text-based online discussions.
A good example of this process was mentioned to me by Ryan Barry, President of Zappi. They have recently installed a system with a global alcohol brand with 800 marketers. The system was vetted by the insights function, but on a day-to-day basis marketers can test concept, ads and such, without needing to consult an agency or the internal insights team.
What does all this mean for market research?
These changes raise a number of questions, such as: What will happen to quality? What will happen to market research agencies? What will happen to client insight teams? What will happen to the trade associations? What will happen to market research ethics? What is the future for the term “market research”?
I will now try to address each of these questions.
What will happen to quality?
My expectation is that the quality of the median project will improve year on year for the foreseeable future. When people talk about the quality of automated, platform-based research, they tend to compare it with the best of human-created research, and that makes the automated option look like “dumbing down.” However, the real impact of automation is not on the experts, it is on the growing number of non-experts. For the under-trained and badly trained, the research will be much better if it uses templates, methods, samples, and reporting protocols that have been created by experts with reference to best practices.
In many cases, the best research will be bespoke, crafted by humans, but in these cases, it will be slower and more expensive. The bespoke options, the specialist options, will be used when the complexity and importance of the problem calls for it.
What will happen to market research agencies?
Agencies will continue to grow in absolute terms, with the sector seeing a growth of 1%–3% a year (as it has for the last few years). But this will mean that each year the agencies are a smaller and smaller proportion of the insights’ ecosystem. The growth of the insights’ ecosystem means that there is a lot more money around, and this has been fueling a major growth in acquisitions, something that will continue for the next few years.
The core business of research agencies will become focused on areas that can’t currently be automated, clients that are reluctant to embrace platform solutions, strategic consultancy, and more advanced forms of ethnography and analytics. Storytelling is going to be increasingly key for agencies, the ability to provide something that platforms can’t.
What will happen to client insight teams?
Three quite distinct changes that are happening:
Some teams are shrinking, caught in a pincer between the straightforward research being democratized throughout the organization, and specialist work being performed by agencies.
Some teams are becoming assistants to the rest of the business, mere order takers. These companies have subscribed the platforms and have access to sample sources, the business units send their survey requests and discussion guides to the insight team to code, deploy, and report back.
However, some teams are becoming proactive, becoming coaches, consultants, and change makers. Anybody interested in following this route should read James Wycherley’s new book “Transforming Insight—the 42 secrets of successful insight teams.”
What will happen to the trade associations?
The trade associations, for example, the MRS, Insights Association, and ESOMAR, need to adapt to a world where “market research” is a minor part of the wider insights’ ecosystem, and a world where there are no clear boundaries. We can see evidence that these associations are already seeking to adapt to this new world. The Market Research Society became the MRS and its website barely mentions “market research,” focusing instead on the role of evidence and data. The Insights Association has pinned it flag to the insights mast, and the Canadian Research Insights Council seems to shun the word “market.”
Key roles for the trade bodies will be advocacy for evidence-based decision making, the setting of standards, education, networking, and the promotion of business opportunities.
What will happen to market research ethics?
Two key tenets of market research for the past 75 plus years have been participant anonymity and the separation of the commercial aspects of a business (e.g., selling and marketing) from research. Both of these shibboleths are being questioned. Most insight professionals see the benefit of integrated data sets, creating 360° pictures of customers. However, this means moving away from purely anonymous data to a reliance on gaining “informed consent” for the ways in which data is used. Market researchers who insist that their data remain separated from the bulk of a company’s data will find they are increasingly side-lined. Similarly, being integrated into the wider data ecosystem means that data will be used to shape sales. When researchers collected the views of 1,000 customers out of 10 million, it did not matter if they were kept separate from selling. But if automated bots have been used to gather 2 million responses, that data is going to be used for commercial purposes.
As a consequence of the research crossing the twin Rubicons of anonymity and selling, social research will need to separate itself more clearly from commercial research.
What is the future for the term “market research”?
There have been people saying that the name “market research” was no longer fit for purpose for years, but over the last 3 or 4 years this has become an ever-louder shout. Data analysts, design thinkers, UX researchers, and CX professionals often stress they are not “market researchers.” They see market researchers as focusing on “the market,” as opposed to their narrower and people-centric focus. Note, it does not matter if they are right or wrong, most people in the insights’ ecosystem do not call themselves market researchers, and that is not likely to change. The problem is even more pronounced in some other languages, for example, in France the term “étude de marché” is even more focused on the market and is less attractive to any discipline focused on being human-centric.
The term market research will decline, not quickly, but steadily, indefinitely.
In summary
Research is being democratized. This is happening because companies want agile research where the voice of the customer is integrated throughout the organization. This change is being facilitated by the rise of platforms. More research is being done internally, more research is being done by non-researchers, which is facilitating a growth in the use of research and of evidence-based decision making.
Research will grow and its impact will grow. But it will increasingly not be called “market research” and it will increasingly be conducted by people who do not call themselves “market researchers.”