Lucy Davison of Keen as Mustard Marketing presents a summary of the recent BIG polling event…
On 19 May we gathered at the office of Research Now to pick over the entrails of the 2015 UK polling debacle. With Kerian Pedley (pollster, writer, Research Director GfK) and Dr Nick Baker (Partner & Group CEO Quadrangle, PhD Birmingham, BPC/ MRS Inquiry) on a panel ably chaired by Dave Skelsey of Strictly Financial, we wondered what did the oracles have in store for us? And what is MRX actually doing about the highest profile event to have occurred certainly during my time in the industry?
The Independent BPC/ MRS Inquiry was set up after the May 2015 general election, with a team of eminent statisticians looking into the performance of the polls. Said enquiry took a close and academic look at the data, over a year. At the BIG polling event we discovered two things.
- We asked the wrong people
- We're not going to be able to do anything different next time. In fact, it might get worse
We got a prediction (to be taken with a pinch of salt, bearing in mind how poor the industry is at prediction, pace the polls) that we are heading for further problems surrounding the imminent Brexit polls.
So to take these things in turn, the panellists explained that polling is, well, hard. Most research is not assessed and judged in this way, and it's very difficult to get a representative sample of voters, no matter what methodology you use. Young voters build a bias to the centre/ left by getting a larger share of voice and it's hard to get a representative spread of voters over 65. This is getting worse with an ageing population. Plus the actual mistakes of 2015 were not that bad, certainly within statistical variance from previous polls. The problem was that the chosen winner was wrong. They backed the wrong horse.
We discovered that there are some research industry issues that make it worse. Polling is often done for a very low fee or even for free by those agencies who are keen to get their name out there. So the agencies put juniors on the job, often working outside their normal hours for no/ low pay. Journalists expect speed and accuracy but are happy to settle for speed given their deadlines. Whereas the best polling takes a lot of time and money to ensure true random sampling and the right quotas, the media won’t pay for it and would just do their own 'private' polls if polling agencies stopped.
The fact is that although it is only 3% of the research industry, polling is 90% of its public profile. I think the reason we are a little complacent about the results or what action to take is that our typical corporate research clients have simply carried on as normal, and not suddenly got all worried about data quality. They have always worried about data quality and it remains a really big issue for the industry as a whole. As Nick Baker said, 'The population of people who answer surveys is relatively small – and a bit weird'. That is far more of an issue, and the polling results only shone a light on it. The core issue is that as an industry we rely on a shrinking pool of respondents, do not incentivise them well enough and present them with poorly designed surveys they don't enjoy doing. We are driven by cost cutting to find the cheapest sample, but you pay for what you get. No wonder big data is the panacea it is presented as, but big data won’t work for polling.
So what to do? In my view there should be one source of approved polls paid for centrally by Government with clear and demonstrable independence, with decent budgets and using decent samples. The journalists can then spin what they get from this as much as they like. At least there is a chance it will be more accurate. And given new developments in predictive markets and behavioural economics are there not new methodologies we could be using to get more interesting and accurate results?
I think the MRS and BPC should engage with the Government to set this up and police it.
What do you think?
Thanks to the Business Intelligence Group for a really entertaining meeting that really got to the heart of the issue. Roll on 23 June. Or not?