A danger that “data-driven” organizations face is confirmation bias, the tendency we all have to cherry-pick information that confirms our existing beliefs or ideas.
Consider a software gaming company looking to optimize its advertising spending. The head of marketing has an intuition: in-theater advertising will boost sales. The marketing analytics team proceeds to do an optimization analysis that suggests that shifting some ad spending from online-video (such as YouTube) to in-theater advertising will produce the desired outcome.
Right before the holidays, the company executes the new plan. Sales of new version of its flagship game increase in 23% compared to previous versions. The leadership team celebrates the victory and decides to continue with the new budget allocation that favors in-theater over online-video advertising. At the end of the next quarter, both the ROI on advertising and the company market share have significantly decreased.
What went wrong?
Once the boss came up with the idea to invest in in-theater advertising, the entire team started looking for data to confirm that hypothesis, ignoring any information that rejected it.
Instead of doing a local experiment and using the results to refine its calculations, the company forged ahead with the new strategy. Then, after the first alleged win, the team didn’t look beyond the increase in sales to weed out misleading or false explanations. If it did, it would’ve found that a) the main competitor had a 40% increase in sales during the same period, suggesting that the positive result was the consequence of “a rising tide lifts all boats” rather than the use of in-theater advertising; b) the effect on sales of online-video advertising (now significantly reduced due to budget reallocation) was much greater than believed.
Three strategies to avoid confirmation bias
1) Focus on learning
It’s fine to start from an idea that is initially based purely on somebody’s hunch. The danger resides in cherry-picking data that reinforces that initial belief.
When working with big data sets, it’s easy to find statistically significant relationship between random variables. As the phrase famous in the statistical literature goes,
If you torture the data long enough, it will confess to anything.
This is why, after an intuition for the “next big thing”, the initial steps need to involve generating hypotheses and conducting experiments to validate them. The cycle of sensing, analyzing, and discovery may need to be repeated many times, ideally in cycles of short duration.
2) Look for disconfirming evidence
When we come up with what looks like a great idea, looking for supporting evidence is par for the course. Equally important, but often neglected, is the process of looking for evidence that proves us wrong.
Hypothesis maps are useful to capture both types of evidence and how they relate to each of our hypotheses, helping us reach appropriate conclusions.
3) Use premortem to learn from the future
Premortem exercises are a powerful tool to challenge assumptions and seek disconfirming evidence. In a premortem, instead of waiting until the end of a project to find out what went wrong and learn from the future, we go on an “imaginary time travel” to avert real failures.
To do a premortem, block some time at the beginning of a project to imagine a time in the future after it has been completed. Assume that the effort had the worst possible outcome. Write a story about why the failure occurred. Do the same with the opposite result: pretend the project was a roaring success, and write the story that explains why.
Evidence suggests that looking at what could go wrong using the premortem technique to think in terms if “did this, not that”, rather than “do this, not that” give us a different, superior vantage point.
Awareness of the problem is the first step to overcome it
Confirmation bias permeates our personal and professional lives, and can be extremely difficult to overcome. No one likes to admit they’re wrong, so our first impulse will always be to try to find evidence that justifies our beliefs.
The best decision-makers, however, are experts at avoiding confirmation bias. They focus on learning and acquiring new data. They look at both confirming and disconfirming evidence, balancing the persuasiveness of one against the other. They use premortem exercises to learn from the future.
By using the same three strategies for your own ideas, and ensuring that anyone defending a solution use them, you’ll dramatically reduce the risk of making a bad decision as a result of biased analyses.