Presenting a rating scale

07 Mar 2018 | Research & Business Knowledge

Alison Lyon recently faced a problem which was outside of her comfort zone.  But the power of the egroup came to the rescue… here is a summary of the advice given.  Many thanks to all the generous souls within the ICG community who continue to offer their help.

Context:  The client wanted 16 (very similar) products rated on a 10 point scale and I had 28 participants.  How to get something meaningful which didn’t pretend to be proper quant?

The solutions:

  • if it’s appropriate, put the data into an NPS (net promoter score) scale (Survey Monkey now offer this for a fee) (very appealing to me as a qualitative researcher but not really getting down dirty with the data)
  •  first look at the distribution of the scores across the 10 points: are they even, clustered in the middle, or mostly at the extremes (Marmite-style).  This will show you a) how best to display the results to show the true picture of reactions (a simple mean vs showing the clustering/ unusual distributions) and b) what you need to investigate (e.g. one product caused love-it-or-hate-it responses, whilst another created not-bothered-or-hate-it) to explain/ hypothesise
  • if there are only straightforward distributions then plot the mean (and/ or modes and/or medians to check for any variance between average scores) scores out of 10 and show in increasing or decreasing order (If you’re a show off and have a large enough sample, you can calculate ANOVA and post hoc comparisons to show significant differences)
  • put the results into a stacked bar chart (and even group some of the responses so that you have a scale which will visually be more meaningful – either thirds or fifths), this will highlight the variation in distribution, then concentrate on the ratings which stand out (either negatively or positively)
  • try to make the charts more accessible and meaningful e.g. group the products meaningfully (format/ price/ flavour), show the most highly rated on one chart, the least on another, or does the client themselves have a preferred way of grouping the ratings (e.g. combine scores 1&2, 3&4, 4&5 etc.)
  •  if it’s a straightforward preference question and distributions are normalise,  rank by choosing the top one or two boxes rather than the mean

There were other options:

  • an internal preference map should there be lots of significant differences.  This is a principle component analysis with consumers as a factor.  This would show the direction of liking of each consumer and with respect to the products.  If there are enough consumers (I had 28, so no, there weren’t) you could even look at segments.
  • or if there were enough data points you could look at range optimisation via TURF analysis
  • and a reminder to show what workings I can: sample size, confidence limits, original question etc.

ICG quanters, I salute you, you are awesome! 

Alison Lyon