Posted by Ray Poynter, 5 August 2019
One of the questions I am often asked relates to the challenges facing market research and insights. The first thing to highlight is that the world of market research and insights is a very diverse group, including leaders of client-side insight teams, leaders and entrepreneurs from research agencies, leaders of suppliers to the research industry (e.g. panel and software companies), and employees of all of these (ranging from neophytes to veterans). Some of the key points are the same for all of these groups, but for some situations the emphasis is different.
Here are six key challenges.
- The need for speed
- Curation
- The decline of science
- The human dimension
- What AI? When? How?
- New business models
The Need for Speed
Most of the projects that organisations undertake are not research projects. They are projects where research can provide help, but they are not research projects. A new product is not a research project, the management of 160 stores is not a research project, and optimising the media mix from a new advertising campaign is not a research project. The speed of business is getting faster; agile product development, real-time, big data integrating with the management of stores, and programmatic advertising is taking the human out of the process. If research is going to expand its role, to promote the use of evidence-based decision making, it needs to operate at the speed of business.
Things that will help deliver increased speed are standardisation, automation, AI, and new business models.
Curation
There is too much data to handle in the ways we used to handle data. The data streams that are available today are not being adequately blended to maximise their usefulness. There are two key aspects to curation:
- Reducing the size of information presented to users. When the size of the data doubles, the size of the delivered answer should normally not increase. When the data becomes 1000 times bigger, the size of the answer should remain short and succinct.
- Drawing information from different sources to provide a clear and succinct answer. Note, blending data should not result in a bigger answer, just a better answer.
Curation is one of the key skills that needs to be taught to insight professionals. Curation is the art and the science of extracting answers that help businesses make better decisions. There is also a need for a major change from the suppliers of analytic and visualisation software. At present most software providers highlight how much they can produce, but they need to find a way of highlighting how little they need to produce to provide the relevant answer. One of the interesting developments in AI is the growth in summarising techniques (including topic modelling).
The Decline of Science
There seems to be a decline in the confidence in science and an increase in the acceptance of opinion. The rise of interest in the Dunning-Kruger effect is perhaps a symptom of this change. It is clear that there are many leaders, followers, and others who feel their views on climate change, the impact of trade tariffs, and the economic consequences of Brexit are equally valid to those of the experts.
The challenge for insight professionals is that evidence from research that contradicts the views of stakeholders is likely to be ignored. This change means that the rise of tools such as video and storytelling even more important. Insight professionals may choose to use science to find their answers, but they need to utilise emotional techniques to convey their results.
The Human Dimension
In the rush to big data, automation, and AI, the human dimension is becoming increasingly important. The more we know what people do, the more we need to know why they do it and what interventions might cause people to change what they do. Market research has always focused on understanding people, and since the 1940s has sought to do that by utilising both qualitative and quantitative approaches. Key elements of this human-centric focus will include:
- Establishing what the business question really is.
- Defining a research question, something that research can find out that will help answer the business question.
- Understanding the research results. With Big Data, AI and automation the calculation of the results will increasingly be a task for data scientists and bots, but the interpretation requires human insight.
- Finding the story in the results.
- Communicating the story in ways that lead to action.
What AI? When? How?
Automation is now a mainstream component of market research and insights, but there are a large number of questions about how and when AI is going to be deployed more widely in market research. We are seeing some early steps in areas like panel management, the coding of open-ended comments, speech-to-text translation for video, sentiment analysis, and chatbots. However, none of these have made major changes, yet. For example, companies not using AI are still able to compete.
Coming back to the issue of speed, I think the key developments will be those that facilitate faster research. I foresee three levels of AI
- Things which optimise processes (data cleaning, routing, management etc)
- Things which improve the scope of DIY tools – so non-experts can do more, and do it quicker and safer.
- Things that will enable power users to do more. One of the key shortages in the future will be in the area of experts (including data scientists, business analysts, and ethnographers) – tools that make experts more productive (i.e. faster) will be valuable.
New Business Models
The traditional research model was that a client would speak to a small number of agencies, and issue RFQs. The agencies would bid for the project, centring their timeline and costing on the data collection process, and one agency would be appointed. The project would then be conducted and debriefed, with the focus of the presentation being the research that has just been conducted.
Clients are increasingly finding this business model, based on commissioning data collection, does not work for them. Data is cheaper, data is plentiful, data sources need blending, and the value is in the answers, not in the process. Models with low marginal costs (for example a simple, automated project from a self-serve portal), or a DIY project, or a project that is part of a subscription (e.g. an online community) are all more attractive than the traditional model. Perhaps other new models will appear, for example, perhaps somebody will get the market place model to work. Or perhaps a gig economy system where an app or company sits between end clients and large numbers of skilled freelancers will evolve. But I think it is clear the old business model will decline.
Advice to Young Researchers
Whilst the picture is different for different people, the group I am most often asked about are the people new to the world of market research and insights. Here are my thoughts specifically for them.
- Understand business – for example, how does your client make its money?
- Understand people – even data scientists need to understand motivations and psychology
- Be T-shaped – have a broad understanding of the core skills for market research and insights and have a deep speciality (it could be coding, it could be semiotics, it could be visualisation – the topic almost doesn’t matter, as long as you are good at it and passionate about it).
- Learn how to be fast – keep looking for approaches, tools, methods, apps etc that will allow you to work faster than other people around you.
To read more of Ray’s blogs, visit here. https://newmr.org/