The REAL Problem With Polls

With the Scottish referendum over, the polling post-mortem begins. We’re hearing the usual complaints: they got it wrong, they overestimated one side, and prediction markets were better.

Since we have a very successful research product based on the fact that predictive markets work better than asking intentions, you’d expect us to join in this pile-on.

But a lot of the criticism of polls is based on a misunderstanding about what they actually are. Continue reading

The Voice Of The Crowd

You might have heard about BrainJuicer’s mention in the Economist, but how about our recent appearance in OK magazine? Not for adorable photos of the office dog (that particular bidding war rages on) but for a Predictive Markets study our German team ran on reality TV pop programme The Voice. We forecast a win for a guy named Andreas – and that’s just what happened.

We’ve had other successes picking Reality TV winners with Predictive Markets, but a few misses too. The structure of reality TV means that there are multiple decision moments – one whenever the public votes. Each of them can be framed by producers hoping for specific outcomes and twists in the storyline. For instance, in the German Voice, our predicted second-place contestant, Caro, went head-to-head with Andreas earlier in the series and – as we would have predicted – was eliminated. Continue reading

Who’ll Win The Oscars? Don’t Ask Me!

Happy Friday! Here’s some punk brass band music from Germany.

Brian, you may be asking, why are you showing me this… this thing?

It’s all to do with the Oscars, naturally.

At first sight the Oscars looks like a brilliant opportunity for a bit of cheeky research PR. Get your preferred testing technique, use it to call the winners right, and watch the Retweets roll in? But it’s not that simple. The problem with Oscar Night is that the winners aren’t decided by the public – they’re decided by a small coterie of experts, which makes them extremely opaque to the research gaze.

Take, for instance, our horn-y German punks. They entered their song for the heats of this year’s Eurovision Song Contest, bidding to become Germany’s official entry. The results were decided by a combination of radio and television public vote, and by an expert jury.

So what happened? The public loved them – they swept radio and television votes.

The experts put them last. No Eurovision for you, punky brass guys!

This is the difference, in a nutshell, between box office and expert judgement. Market research is in a good position to predict box office – when BrainJuicer pitches its Predictive Market testing tools, we often lead with self-funded studies on Hollywood and Bollywood films, which picked hits with impressive accuracy. But we wouldn’t use Predictive Markets on the Oscars, because “experts” are, frankly, too capricious.

People can and do speculate on what experts will choose – it’s what makes awards shows fun. And you could, if you liked, survey a lot of those people and aggregate their ideas about what the experts will do. Couldn’t a market researcher do that? Sure – but there already a bunch of people whose job it is to aggregate popular ideas of what will happen, and we don’t call them “market researchers”. We call them “bookmakers”.

Door No.24: Toy Story

Our Advent Calendar of Experiments comes to an end…

The Experiment: The final door on our Advent Calendar opens and behind it is – a pile of toys! But not just any toys. These are the toys identified by UK toy store Hamleys as the runners and riders in the race to become Britain’s top Christmas toy.

There’s tablet-for-kids the Leappad2. There are mighty brand team ups – Lego and the Lord of The Rings. There are old favourites making a nostalgic comeback – Cabbage Patch Kids and Subbuteo. There are newcomers – online catch-em-all sensation Moshi Monsters. There are beloved characters – Spider-Man and Barbie.  And there is the bizarre and distressing spectacle of Mickey Mouse dressed as a breakdancer.

You’d never guess he was 85.

We tested all these toys using our Predictive Markets method – harnessing the wisdom of crowds to pick a winner. Would we agree with the bookies? Would we predict the actual winners? Read on… Continue reading

Door No.18: PM to AM

The 18th of BrainJuicer’s Advent Experiments…

The Experiment: For a research company, improvement is as important a goal as discovery. Experimenting to find new ways of doing things goes hand in hand with experiments to find better ways of doing things. This is especially important as “new” can sometimes be seen as “risky”, “expensive”, or “difficult”.

So at BrainJuicer were particularly excited at the idea of experimenting with one of our most established and successful solutions, Predictive Markets, with the goal of seeing how far we could improve the turnaround time. Predictive Markets use the “wisdom of crowds” to find successful concepts, and we set ourselves a simple goal: we wanted clients to be able to launch a test one day (in the PM) and get the results the next morning (in the AM).

The specialist in overnight delivery.

Could we achieve this Santa-like speed? Continue reading

Door No.14: A Predictive Christmas Market

The 14th entry in our Advent Calendar of Experiments, and the first one which is actually seasonal…!

The Experiment: Validation is a vital part of market research innovation – if your results aren’t replicable and applicable across markets and categories, your new idea might not be so useful. We’ve been running Predictive Markets as a concept testing tool for several years now, based on the ideas in James Surowiecki’s classic The Wisdom Of Crowds – a ‘crowd’ of independent predictions is likelier to get to the truth than a recruited group of experts.

Of course, the problem with concept tests is that you usually only see how well the good ones perform. So we were excited but also nervous when a client came to us with an idea for an experiment – test a bunch of concepts they were definitely going to launch and then see how well all eleven performed in a real test market.

“Are you telling me my concept is a turkey?”

The category? Christmas food. But would it be “ho ho ho” or “no no no” for predictive markets? Continue reading