Diversification Bias is one of my favourite behavioural economics concepts. Basically, it’s the idea that if you predict your choices in advance you build in a lot of unrealistic variability.
For instance, if you were to ask me “Brian, what bow ties will you be wearing this week?”, I might answer, “Oh, the red one today, then the blue one tomorrow, then the green one,” and so on. But when it comes to the bow tie crunch – the actual moment of decision – I have to admit I always pick red.
A better predictor than you are.
Why do we predict our behaviour so badly? Partly it’s self-deception – we think we need more variety than we do. Partly it’s social pressure – if choosing films to watch, for instance, we will build in more highbrow choices if picking them in advance.
And it’s the world of films which brings us a great example of how our inability to predict our preferences affects real-world business decisions. A few years ago Netflix, the DVD rental and streaming video business, offered a million dollar prize to anyone who could improve its recommendation algorithm – the thing which looks at your past choices and suggests future picks. A bunch of scientists won the prize and got a lot of publicity for it.
But here’s the thing – Netflix never used the winning algorithm, even though it did make more accurate recommendations.
Why? The winning algorithm predicted rental choices – which are made in advance of viewing, not in the moment. And it turned out it didn’t predict streaming choices – made in the moment – as well. In other words, it predicted what people thought they would want to watch, but not what they actually wanted to watch. And the difference between the two meant that Netflix never implemented their prize-winning algorithm.
So even with just one choice, what people said they’d do and what they did turned out to be crucially different.
For more on diversification bias and its implications, here’s a fascinating blog post by Alain Samson, Brainjuicer’s Scientific Advisor.