The 14th entry in our Advent Calendar of Experiments, and the first one which is actually seasonal…!
The Experiment: Validation is a vital part of market research innovation – if your results aren’t replicable and applicable across markets and categories, your new idea might not be so useful. We’ve been running Predictive Markets as a concept testing tool for several years now, based on the ideas in James Surowiecki’s classic The Wisdom Of Crowds – a ‘crowd’ of independent predictions is likelier to get to the truth than a recruited group of experts.
Of course, the problem with concept tests is that you usually only see how well the good ones perform. So we were excited but also nervous when a client came to us with an idea for an experiment – test a bunch of concepts they were definitely going to launch and then see how well all eleven performed in a real test market.
“Are you telling me my concept is a turkey?”
The category? Christmas food. But would it be “ho ho ho” or “no no no” for predictive markets?
The Results: Thankfully for us, the crowd came through. But it wasn’t easy.
Initially we got quite a poor correlation between our predictions and the results. But then we hypothesised that the way we were showing the concepts wasn’t reflecting the way they were displayed on shelf – some of the products had particularly innovative or prominent displays, while others were less well served. So with the client’s help we used 3D imaging to give a better impression of the display context.
This made all the difference – a 0.2 correlation between sales and performance across the range became a 0.7 correlation when we included the display context. Which isn’t perfect, but is excellent for a tool designed to screen at the concept stage. Even in the run up to Christmas, the crowds are still wise, and we have more evidence that predictive markets work well as a way of identifying performers of whatever quality. The lesson is, though, that display and in-store context really matters – something we know from all our behavioural economics work. If you can possibly include those in your concept testing, you should do.
To learn more about predictive markets, check out this webinar showing our latest thinking with them. And keep watching the Advent Calendar for more on PMs, including the fastest PM in history! (We think)