Shock and Awe
Your correspondent started the day buoyed by the thought of joining his industry colleagues in debate and discussion at the annual MRS conference. It had been a while since I’d attended a full-day’s papers, the last time being back in the dark ages when the presentations seemed to be a series of adverts for agencies and not much more.
Impact 2015 promised much more – Minds, Meanings and Metrics – Consumer Understanding in an Uncertain World (no less).
The keynote speech comprehensively torpedoed this mood of optimism; not because it wasn’t good but because it made the listener wonder why they had spent any time pondering the vagaries of sample design and projective techniques, instead of building a nuclear shelter and stocking it with their favourite non-perishable foods and DVDs.
I love listening to senior military personnel speak. Their clarity of thought and certainness in delivery inspires confidence and admiration. General Sir Richard Shirreff, formerly NATO Deputy Supreme Allied Commander, was no exception to this rule and delivered a fast-paced résumé of the current geo-political threats to the ‘West’.
This pretty much boiled down to an increasingly twitchy, boxed-in and nuclear-primed Putin and “the hydra of Al Qaeda” including ISIS (so-called or otherwise) and its allies and off-shoots, which seeks to establish “an arc of instability” across Syria, Iraq and Lebanon and much of Northern and sub-Saharan Africa.
Ultimately, he conceded, these were political rather than military issues, that required the “stabilising of the unstable” and “removing the soil in which terrorism flourishes”.
Sir Richard framed much of this in a discussion around the concept of strategy, strategic thinking and understanding the strategic environment. Here was the link to his audience – we were in the same business. Strategy, whether commercial or military, is essentially adversarial and relies on “integrating the complex calculus of ends, ways and means”.
These, however, are such shifting sands that they require a deep understanding of the strategic environment and proactive and dynamic management – in other words, comprehensive intelligence gathering and fast and accurate interpretation.
Above all, he said, this needed to address the “human environment”, make “no assumption of rational behaviour” and recognise that political leaders’ decision- making was often emotionally based (see Putin above).
So, there we had it – an uncertain world indeed and we’d arrived at the conference worried about declining response rates and the representativeness of research panels! (more of that later).
Sir Richard’s advice, in a nutshell
- “In a world lacking clarity/perfection, the best strategic choices are the least wrong ones”
- “Train for certainty, educate for uncertainty” – backed up by a quote from T. E. Lawrence (of Arabia). “The greatest commander is he whose intuitions most nearly happen. Nine-tenths of tactics are certain, and taught in books: but the irrational tenth is like the kingfisher flashing across the pool and that is the test of generals. It can only be ensued by instinct, sharpened by thought practicing the stroke so often, that at the crisis, it is as natural as a reflex”
- Evidence was key to (good) decision-making and perhaps our industry could help (?!), however he bemoaned the fact that currently, “by the time we gain sufficient understanding, our adversary has reformed and evolved again.”
- “Doing nothing is not an option”
Trotsky’s chilling –“You may not be interested in war but war is interested in you”, darkened the mood further.
He finished with a quick overview of one of his other passions, which is Genderforce, an admirable organisation that aims to “fight and prevent acts of sexual and gender based violence in conflict and post-conflict situations”.
Its work includes the gathering of the grim evidence of the existence and extent of such war-crimes by partnering with local organisations that have the trust of the populous, using friendly informers, using open-source data and technical intelligence (mainly satellite-based) and recce teams on the ground. All of which had the sound of something our industry’s techniques and expertise could help with.
After that, there didn’t seem much point in debating .. er .. anything .. really .. but ashen-faced, we proceeded to ‘The great technology mash-up’, which gamely promised to present both sides of the device-driven research v. the humanity/ empathetic base of traditional qual argument. Are friends electric? as Gary Numan once unfathomably pondered in the days before Facebook added some game-changing, digital weight to this question.
Neil Griffiths presented an ad for Blinc (uh-oh back to the bad old days) under the slim pretext of demonstrating that good advertising research is about combining techniques – dial-testing, facial coding, implicit testing and predictive markets – or Consilience (TM) as they call it – and that it’s all about ad optimization rather than fuddy-duddy ad-testing.
Paulo Panizzo (BSkyB) – a Blinc client – said his organization was open to innovation and looking into pipeline products and combination products and that they want them to work. He recognized that the world was a complex place and that combination techniques had the potential to “add layers of truth”. However, his endorsement of Blinc’s work seemed less than ringing in a ‘the jury’s out’ kind of way and he conceded that the vast majority of his research spend continued to focus on more tried-and-tested methodologies.
Ben Scales (davies+mckerr) demonstrated how Heineken used everyday tech solutions to bring to life their target audience (25-35 year-old males) and how they “connected with the city”. This inevitably revolved around the use of social media and smartphones and (the somewhat disturbingly coined) ‘self-nograghy’, whose linguistic roots can sometimes appear closer to the por- rather than eth- variants.
While sharing photos and video-blogs in a research setting isn’t ground-breaking, this mash-up of digital and device-based information gathering did seem particularly pertinent to this target audience and the use of tools that ‘they know and love’ made the data gathering efficient and cost-effective. It also helped the client to watch and share the journey and bring the research to life for a group beyond the core project team.
Crucially, he concluded that these data gathering techniques required the same time and thought as traditional methods – in other words, the interpretative basis is still a human and empathetic one.
Rob Ellis (COG Research) highlighted the unreliability of self-reporting and the growing portability and scalability of some technology-based data gathering. In particular, he showed a combination of on-the-go eye-tracking and skin response measures (galvanic skin response or skin conductivity response depending on your preference) in a supermarket shopping scenario.
Thus, arousal (in a general, rather than por-nographic sense) can be linked to whatever the person looks at and creates “a new, richer context for finding out why (.. people behave/make choices/etc).” However, this still requires the interpretive and time-consuming elements of traditional research – do we have a theme developing here?
Apparently, the dial moves between around 25% and 160% of normal skin conductivity and this participant was much more aroused while looking at their shopping list than by any of the in-store displays.
Another eye-tracking/skin response mash-up concluded that participants/ collaborators (we’re not allowed to call them respondents in the new digitally empathetic age) were much more attentive to full screen i-pad/tablet ads than those on Youtube, where all the extraneous screen guff is very diverting. No sh*t Sherlock.
Rob had the good grace to admit some slight nervousness around the circularities involved in “watching people, watching people, watching television” on a project that included simultaneous eye-tracking, skin resistance monitoring and videoing of Gogglebox viewers!
Peter Trotman (Jigsaw) took on the role of Canute-like technophobe and gave an impassioned plea for traditional quallies to “walk tall in the land of machines”. He implored us to define our job in terms of emotional intelligence and told us that our job – i.e. qual researchers – was still one of a few that could not be done by robots. (Though some of us may have witnessed some things distressingly close to that in our long careers!)
We must stress the relational elements of our work, its intuitive and empathetic nature and that we’re in the business of revealing insight not producing data – almost like a doctor asking clients to “relax, you’re in good hands”.
Something “happens in the room that isn’t verbal, might involve counter-transference and/or mirror neurons and it’s unique in a way that machines can not mimic.”
Therefore, we must guard against a rising obsession with innovation, the lure of scientism and the love of technology.
Well good luck holding that tide back, Peter! The previous papers seemed to suggest that technological solutions are here to stay and have some elements that open up the black box of qual and make them more demonstrable and client-friendly.
Each of the technologies demonstrated still required interpretation and the skills of experienced researchers, ethnographers and quallies to unpick. The real danger seems to me to be, not in us embracing these technological innovations and data gathering tools but in the possibility of clients thinking that you don’t need that interpretive element and experience in conjunction with them.
Maybe ‘twas ever thus’, in that clients might think that having a chat with a few customers is qualitative research.
I wanted to split the next session between ‘Solving social problem through research’ and ‘Taking a walk on the wild side – how does research respond to newness’. The latter because our very own Tom Woodnutt (Feeling Mutual) was the final speaker but I also wanted to hear about GfK’s work with Citizens Advice.
Sadly, these papers were annoyingly rescheduled to be against each other and I belatedly plumped for walking on the wild side. The first ‘paper’ was engagingly delivered by James Eder, a young chap who appeared to have swallowed every self-help and management book available and blended the contents into his own personal and evangelical entrepreneurial doctrine.
Powered by one-liners and a complete disregard for the benefits of MR, he’s a force of nature and has built a very successful digital business (The Beans Group feat. www.studentbeans.com ) from scratch. Terrific and good on him.
His approach to MR is “why bother with that?” He’s happy to take his ideas direct to market and sell the pants off them. If that doesn’t work (which didn’t seem to happen to him very often), then do something else. Be an entrepreneur not a ‘wantrepreneur’ – i.e. someone like 99% of his audience who sit around in pubs saying ‘what we ought to is ……./what the customers really want is ……’ and then forget to do anything about it the next day. Just get out there and do it and don’t take no for an answer!
However, 99% of this conference audience also work in the now utterly pointless MR business and had probably been relatively happily plotting a career-path within it, up to this point – so, barring some sort of Damascene conversion and though undoubtedly personally inspiring as he was, probably didn’t find these thoughts particularly helpful to their current working lives.
In ‘The knew new: innovation through familiarity’ Nick Gadsby (Lawes Gadsby Semiotics) presented his thoughts on newness and how most successful NEW! is based on .. er .. something old.
The theory he expounded is that it is part of the human condition for us to be pattern recognisers, so when faced with something new we look for comparisons and cultural analogs. Thus evolution is a comfortable idea whereas genuine innovation can be too unfamiliar.
He identified three types of newness:
- Incremental newness – that builds on, continues the development of anything that’s gone before, e.g. the evolution of the Windows logo
- Oppositional newness – that does the exact opposite of what’s gone before, e.g. Coca-Cola Life and it’s green logo
- and Archaic newness – that acknowledges that the past is a great source of newness, e.g. the recent re-emergence of vinyl record releases, beards, bicycles and craft beers
This was engagingly delivered and introduced something called the Max Factor beauty calibrator – a 60 or 70 year-old device that looked like some sort of sado-masochistic, iron-framed, punishment helmet .. or scolds bridal, as one or two of the better-informed audience suggested. It turned out that it was used to measure women’s divergence from the beauty ‘ideal’ of the day and, of course, recommend Max Factor remedies, no doubt.
I tried hard to find more than an oblique relevance of this paper to our industry but failed. It might just be me.
Thankfully, Tom Woodnutt (Feeling Mutual and ICG member – ta da!) rescued the session with some good thoughts around a manifesto for change in the entrepreneurial era.
This addressed some of the issues raised by James Eder earlier, in terms of how we prove our value to entrepreneurs. Big business is becoming more entrepreneurial, a bigger proportion of clients will have initiated their businesses in the fast-feedback digital age.
I won’t do the full presentation justice and I’m paraphrasing madly but his emphasis was on establishing a new research paradigm and away from the traditionally expensive, slow and broadcast model based approaches that tend to be narrowly defined and controlled by the client’s agenda.
This new paradigm would see researchers:
- moving from messengers to value negotiators on behalf of the customer/consumer – listening outside the research, collaborating with customers
- shifting away from the ‘obsession with science’ towards something more experiential where the content inspires engagement rather than the provision of incentives
- becoming more curator than collector where there’s less emphasis on data collection/sample size and more emphasis on data-quality and richness and effective communication of the discoveries within research budgets
Is this feasible? Is this the sort of thing that good research always tried to deliver? Is it really deliverable? I don’t know but it was refreshing to hear these ideas, as well as something with a bit more direct relevance to the future of our industry and the aims of the session. Well done Tom!
A nice lunch catching up with colleagues old and new was followed by ‘The Panel: The client verdict. What do they really want from research?’
Sadly, I’m not much the-wiser beyond the old feeling that largely what they want is instant justification of the decisions they or their bosses want to make anyway.
I found this session both depressing and alarming at the same time. We must bow down to ‘Big data’ which has freed us from the tyranny of research, we must ignore the opinions of anyone not digitally able to express themselves (immediately), we must stop taking time to deliver robust, thought-through, interpretive research and instead concentrate on instantly divining meaning from the bleatings of avatars and alter-egos and all the zeros and ones that digital commerce can throw at us.
Forgive me. They didn’t say that but I struggled to find a positive word for our industry’s long and successful history of guiding clients to better decision-making. Instead, it felt like everything that has gone before was worthless and everything that ‘Big data’ and social media promise is wonderful and unencumbered by the unhelpful overlay of researchers. It was, at last, “the democratisation of research.”
Dianne Thompson, our President, scarcely had a good word for our industry – mocking segmentation as a tool – “They were all very amazing and sexy because we could visualise very well. But in terms of how we could use them, they were useless!” – and blaming research for delaying increasing the price of a lottery ticket up from £1 to £2 for several years, despite that being a business inevitability. Apparently, when asked, consumers didn’t rate that idea very highly. Turkeys not voting for Christmas – there’s a shock.
Sue Unerman of MediaCom could scarcely disguise her glee that the digital world had rid her of the need for research, as she was now able to instantly divine the true path from tweaking digital messages to see which ones worked best. “Designing things, putting them together, seeing how they work is almost anathema to the traditional market research brief”. Really ?!
The poor segmentation industry received its second coruscating critique of the session. One of Mediacom’s clients had, I thought she said 950,000 but it may have been 950, different, targeted text messages at its disposal for different audience members. Apparently none of them people like me, who receive dozens of spectacularly inappropriate and off-target digital messages daily.
To be fair, Anthony Jones (Royal Mail) and Simon Carter (Fujitsu) redressed the balance slightly. Anthony Jones putting forward the view that he wasn’t a fan of speed for speed’s sake, Simon Carter wanting us to understand our clients’ businesses and concentrate on communicating the findings effectively and in a way that helped them filter out to operational areas. All agreed that combining research methods was usually most helpful but few seemed willing to wait or pay for that to happen.
However, anyone not from our industry might easily have watched this performance and thought that it was a celebration of the end of the research industry rather than something aimed at cementing its future.
Thompson added: “I do think the research industry is lagging a bit behind. The very fact that you talk about people as ‘respondents’, rather than having a dialogue with people, needs to change. The whole role of social media has sped everything up”, as if to imply that the research industry hasn’t embraced any of these changes and hasn’t, for many, many years, sought to be the consumers’ voice in a world dominated by marketing speak, policy gibberish and management fad. Dianne, I remind you, is our President.
So, tough luck if you’re still trying to use traditional techniques to good effect and condescending sympathy, if you recognize that your target audiences aren’t all among those who are digitally buzzing. Losers, the world’s moved on.
Now, I’m not attempting my own Canute-like response to the inevitable here. I do most of my research on-line these days. I’ve even recruited and moderated a Facebook group recently, which I found to be a rich data source, a rewarding, human experience and, whisper it, a small cog in a segmentation development process.
However, this particular throwing of the baby out with the bathwater, seems to me to be a classic case of availability bias – i.e. paying greater and unquestioning heed to what is digitally available, rather than seeking a genuinely grounded understanding.
All the speakers in the technology mash-up session earlier in the day had been at pains to point out the need for the softer and time-consuming elements of traditional research, in order to correctly interpret and contextualise and the noughts and ones. Sadly, Sue and Dianne must have missed that one.
In grumpy mood, I went on to the next session – ‘Fit for purpose sampling in the internet age’. Perhaps I should have attended the papers in the other session that covered empathy, humanity and happiness, just as an antidote to ‘The Panel’.
I have to confess that I had to curtail taking detailed notes at this point because the statistical terminology and theories came at the audience at a relentless pace and listening and grasping the main points was the best I could do.
I think the three industry luminaries who spoke were basically on the same page, though there was some interplay between Reg Baker (Market Research Institute International) and Douglas Rivers (YouGov) that hinted at some previous exchanges.
Reg Baker told us that “Probablity sampling is dead” and that, these days, sample “design always involves compromise.” To all intents and purposes, this means that we all now use internet research panels and that we need to be aware of their strengths and weaknesses. Again, no Canute-like holding back the tide on this one. The industry has opted wholesale for their cheaper and quicker charms.
He said that too many people misunderstood Margin of Error (or MOE to her friends) and that suppliers should give health warnings around any sampling exercise. “Samples are like medicine. Every good sample should have a proper label with instructions for its use.” In other words, be aware of the biases in your sample sources and guide your clients accordingly. Exemplary advice but by the time your data has been cascaded through an organisation via the marketing department and/or infographically reimagined on the board’s brand health dashboard, those health warnings may have been lost in the process.
Doug Rivers disappointingly avoided talking about river sampling but reminded us that MOE was potentially misleading and was only part of the story. He emphasized that there was a difference between sample error – i.e. the error inherent in any sampling exercise (because it’s a sample and not a census) – and any estimation of the total MOE.
He argued (I think!) that the sample error is no greater in an internet panel sample than most list-based sample sources given the same sample and universe sizes – so the reported MOEs would look similar, if this was the only consideration. What might be different is the biases within the sample lists but these are often difficult to fully assess, especially if the list or panel is your only start point.
So the gist appeared to be that, we’re stuck with panels and that what we should do is understand their biases better. That may be easier said than done. Apparently, one august statistical body has got itself into a muddle trying to offer guidance as to when MOEs should be published and shouldn’t, which ended up sounding like you shouldn’t publish them because they might be misleading.
However, Reg Baker opined that the research carried out on biases in internet panels (apart from the more obvious digital channel/use of technology preferences), suggested that they were few. Samples from panels and other sources answered most questions in very similar ways.
All of this seemed to make sense to someone like me, brought up on acknowledging the difference between sample error and overall MOE. This additional variance was caused by design factors. In other words, while sample error may explain and dictate that there’s some variance, quite a large proportion of overall MOE would depend on what biases you had introduced into the research process generally and that were inherent in your sample source, specifically.
Corrine Moy (GfK) backed-up the inevitability of internet panel usage with the view that random sampling would be fifteen times more expensive. She described an ideal process of controlling biases as “Fishing and Cooking”.
Corrine was, in my days with NOP, and as far as I know still is, a paragon of statistical virtue. So to hear her hold that fishing and cooking represented some sort of ideal statistical process was a little unnerving. Fishing and cooking seem to have some unwelcome connotations in the world of statistical excellence.
Fishing, she contended meant assessing and controlling biases between the universe you were researching and the internet panellists you could access – e.g. using quotas. Cooking meant weighting/adjusting the data for those known biases. She reiterated that clients and researchers needed to understand those processes when interpreting the resulting data.
She’s right, of course, but do people do that? Do they have the time and tools to properly assess panel bias and allow for it? I don’t think they do. I think most find that too difficult and time-consuming and most clients and researchers alike operate on the heuristic basis that it’s a sample, therefore it’s representative.
The following Q&A was largely uneventful except that Ken Parker, who gave an interesting talk at a recent London ICG meeting, asked whether it was true that something like 85% of interviews were being conducted among 0.5% of the population. This was qualified a little, down to 85% of on-line, sourced-sample surveys but the speakers didn’t refute the possibility.
Personally speaking, Reg Baker’s soothing comments around panel samples in-the-main, giving similar responses to non-panel samples (see above), largely failed to quell the uneasy feeling that stark statistic engendered. So much for the democratisation of research!
And so we came to the final keynote session, an interview with Sebastian Faulks – writer and novelist, author of best-sellers such as Birdsong and Charlotte Gray.
The interview was interesting and wide-ranging and Mr Faulks was affable and engaging. A pleasant way to spend half-an-hour. I felt I had gained some real insight into how novels are written but learned very little that was novel about insight – see what I did there! (don’t give up the day job – ed.)
The theme of the interview was loosely around storytelling and Faulks contended that “Good storytelling should always be in the service of good ideas”, which I felt was applicable to our industry.
Time spent researching, he thought, far-outweighed that spent writing. “The practical thing is to do the research, read it and then shut it in a room”. Dianne Thompson couldn’t have put it any better.
So, what were the main take outs from Day 2 (or the bits of Day 2 that I attended)?
- We’re doomed
- Train for certainty, educate for uncertainty
- The future’s bright and it’s digital .. but you need time, traditional research skills and to be human and empathetic to understand what all this digital and technology-based research is telling you
- Combine techniques, when and where you can persuade clients to fund and wait for that, for a richer context for finding out why we behave as we do
- There’s nothing new under the sun
- We need to be value negotiators and get experiential and go easy with the science bit (unless it’s got dials)
- Learn how to “fish and cook” your samples
- Failing to sell a new idea beats researching a new idea to see whether it might sell
- Don’t ever:
- Call people who respond to your invitations to participate in research exercises, respondents (ever)
- Say you’re a fan of traditional research techniques on Twitter
.. and while I might feel that some contributions were more useful than others at this year’s conference, I’ll leave the final words to Sebastian Faulks :
“No knowledge is wasted, it all goes on the compost heap of life.”