Confirmation Bias

Last modified: July 10, 2019

What is Confirmation Bias

Confirmation bias is the tendency to seek out data that confirm beliefs you already hold. This does not sound inherently bad. The problem is that people do this to the exclusion of data that disconfirms their beliefs. This hinders people from learning new things about the world since they continually reinforce their existing beliefs because of this bias.

In a business context, this means you might be ignoring data that is suggesting that some aspect of your feature/product is not working because you found another metric that seems to suggest that it is working. Confirmation bias pressures you to ignore the negative signs and focus on the positive.

Graphic of confirmation bias

How to Detect Confirmation Bias

There are 3 common signs that confirmation bias is influencing the data you are looking at.

  • Only good news
  • Limited metrics reported
  • Obscure metrics being used

Only good news

If the only news is good news confirmation bias is probably hiding some inconvenient metrics. Any business decision, product, feature, or process change is likely to have both positive and negative consequences. What you want is for the change to have more positive consequences than negative ones.

Limited metrics reported

Usually people present a few metrics because they are trying to be brief and that is desirable in most cases. What they need to be careful of is if in their brevity they are not highlighting important pieces of data that might not fit their story. If they only show one metric in their presentation of the data there is important data being left out.

Obscure metrics being used

Many times we cannot measure exactly what we want so we have to settle for proxy metrics. As these metrics become more and more abstracted our confidence in them should diminish. If people are confidently sharing convoluted proxy metrics they are likely looking for ways to find positive signals in the data.

What to do about confirmation bias

Institutionalized disconfirmation

Institutionalized disconfirmation is how academia addresses the confirmation bias problem. Anytime you want to publish a paper on a finding it typically has to be peer reviewed. So you have many other self interested scientists making sure that what you submitted has sound logic and analysis behind it.

However academia is finding that this level of review is insufficient, especially in the social sciences. Too many professional articles report data that does not hold up in subsequent trials. In fact it has been dubbed the Reproducibility Crisis. Reproducibility is a much higher bar than examining the analysis, it provides more data to confirm what the paper has laid out.

How can you apply this to your company?

The first step is to create a peer review process. Before an analysis is presented to the company have a system in place where another analyst reviews your analysis and checks to see if they reach the same conclusions. If you have the time and resources it may be worth trying to replicate a test run within your company to confirm it’s findings.

3rd party audit

Hiring a 3rd party audit is how government tends to address the confirmation bias problem. They have different organizations randomly audited by a 3rd party (in this case the Government Accountability Office) to see if records are accurate and to determine how the company is really doing. Is money being spent properly, and are they actually achieving the results they claim.

At times, a journalist can serve as the 3rd party who investigates and exposes shortcomings of government agencies and companies.

How can you apply this practice to your company?

Use data analysts to randomly check analysis being shared. If there are any mistakes or misleading charts, have them reach out to who created the analysis and go through the issues together. Once corrected send an update to anyone who was using or viewing that data.

You can also hire consultants to come in and audit what is going on in your company. This is commonly used in the accounting and finance departments during due diligence periods for investors.

Insurance rules

This is one way business tries to address the confirmation bias problem. It does not prevent confirmation bias so much as put a monthly cost on it to make it less risky. It places the burden on the insurance company to carry the cost of a mistake the company didn’t find. However, this comes with additional oversight by the insurance company. They will enforce rules for what the business can and can’t do, and charge the business monthly for holding this liability.

How can you apply this practice to your company?

Create parameters of what people can claim with their data. Restrict certain types of phrases such as anything causal. If they make claims that exist outside of those parameters, it does not have approval to be shared.

Culture

This is another way businesses tries to address the confirmation bias problem. They create a culture in which failure is not expected but is understood and does not cost the employee large consequences. This has been done most famously in Toyota, where any employee can stop the whole production line if they notice a mistake and in Google X where they celebrate a projects failure with champagne and bonuses.

How can you apply this practice to your company?

Emphasize the mission of your company. Show that the mission is more important than feeling good about the success of a product or feature. To make real progress toward the goal we are going to mess up a lot and we need to know how we messed up so we can make better decisions in the future.

Written by: Matt David
Reviewed by: Blake Barnhill , Matthew Layne , Mike Yi

Next – Selection Bias

Get new data chapters sent right to your Inbox