Confirmation Bias

What is Confirmation Bias?

Confirmation bias is the tendency to seek out or interpret data to confirm beliefs you already hold. It does this to the exclusion of contrary evidence.

In a business context, this means ignoring data that is suggesting that some aspect of your feature, product, or business is not working because you found another metric that seems to suggest that it is working.

Confirmation bias pressures you to ignore the negative signs and focus on the positive.

How to Detect Confirmation Bias?

There are three common signs that confirmation bias is influencing the data you are looking at.

• Only good news
• Limited metrics reported
• Obscure metrics being used

Only good news

If the only news is good news confirmation bias is probably hiding some inconvenient metrics. Any business decision, product, feature, or process change is likely to have both positive and negative consequences. What you want is for the change to have more positive consequences than negative ones.

Limited metrics reported

Usually people present a few metrics because they are trying to be brief and that is desirable in most cases. However, it is important to guard against being unethical. An example of this is if someone shows a single metric in their presentation and deliberately omits contradictory data.

Obscure metrics being used

Many times, we cannot measure exactly what we want so we have to settle for proxy metrics. As these metrics become more and more abstracted our confidence in how accurately they represent what we want to be measuring should diminish. If people are confidently sharing convoluted proxy metrics, they are likely looking for ways to find positive signals in the data.

What to do about Confirmation Bias?

Institutionalized disconfirmation

Institutionalized disconfirmation is how academia addresses the confirmation bias problem. Anytime you want to publish a paper it typically has to be peer reviewed. So you have many other self-interested scientists making sure that what you submitted has sound logic and analysis behind it.

However, academia is finding that this level of review is insufficient, especially in the social sciences. Too many professional articles report data that does not hold up in subsequent trials. In fact it has been dubbed the Reproducibility Crisis. Reproducibility is a much higher bar than examining the analysis, it provides more data to confirm what the paper has laid out.

How can you apply this to your company?

The first step is to create a peer review process. Before an analysis is presented to the company have a system in place where another analyst reviews your analysis to see if they reach the same conclusions. If you have the time and resources it may be worth trying to replicate a test run within your company to confirm its findings.

3rd party audit

Hiring a 3rd party audit is how government tends to address the confirmation bias problem. They have different organizations randomly audited by a 3rd party (the Government Accountability Office) to see if records are accurate and to determine how the organization is really doing. Is money being spent properly, and are they actually achieving the results they claim?

At times, a journalist can serve as the 3rd party who investigates and exposes shortcomings of government agencies and companies.

How can you apply this practice to your company?

Use data analysts to randomly check analysis being shared. If there are any mistakes or misleading charts, have them reach out to the person who created the analysis and go through the issues together. Once corrected send an update to anyone who was using or viewing that data.

You can also hire consultants to come in and audit what is going on in your company. This is commonly used in the accounting and finance departments during due diligence periods for investors.

Insurance rules

This is one way business tries to address the confirmation bias problem. It does not prevent confirmation bias so much as it places rules on the company to prevent certain errors from happening. For instance it may require an inspector (3rd party audit) before you can get a policy on a house. Or it will not cover your life insurance if you choose to skydive or engage in other behavior that is unpredictable and risky. These rules can be overly burdensome but they do prevent certain silly mistakes from happening.

How can you apply this practice to your company?

Create parameters of what people can and can’t claim with their data. Restrict certain types of phrases – for example, anything causal. Claims made outside of the established parameters are not approved to be shared and therefore not “insured” by the company.

Culture

This is another way businesses try to address the confirmation bias problem. They create a culture in which failure is not expected but is understood and does not cost the employee large consequences. This has been done most famously at Toyota, where any employee can stop the whole production line if they notice a mistake and at Google X where they celebrate a project’s failure with champagne and bonuses.

This culture helps you to second guess your motives for only showing successful metrics.

How can you apply this practice to your company?

Emphasize the mission of your company. Show that the mission is more important than feeling good about the success of a product or feature. To make real progress toward the goal we are going to mess up a lot and we need to know how we messed up so we can make better decisions in the future.

Written by: Matt David
Reviewed by: Twange Kasoma , Blake Barnhill , Matthew Layne , Mike Yi