Even if a metric was chosen for a new feature or product, important parameters about how that metric will be measured and evaluated might not be set.
One of the things that makes analysis hard is that there are so many things happening at once that might affect the metric you are monitoring. Determining what caused the metric to move is complicated. A good experiment tries to control for as many of the other things that may be influencing the outcomes of your experiment as possible so that we can determine if our new product/feature impacted it.
In a typical business context there are three parameters that you should spend time defining:
Not setting these parameters in the beginning allows people to move these around at the analysis stage in order to find data that makes their product or feature look more successful than it actually is. And that is bad, very bad. We want to limit the ability for people to let their confirmation bias flare up and allow them to interpret data incorrectly. Being strict on these parameters prevents people from changing how they interpret the data after the results are in.
Let’s talk through each of these pieces and some important considerations.
If you are testing out a feature or product with a subset of your user base, make sure they are representative of the user base you are expecting to use the product or feature. Otherwise the results might be biased and the feature or product will not perform how you expect when you roll it out to everyone.
Questions to ask yourself:
- Is the cohort unique in any way?
- Am I picking this cohort out of convenience?
If the answer is yes to either you need to find a different cohort.
People report good news too early and bad news too late. If day one after you launch there is a big spike in the metric you are tracking it is natural to feel excited and send out a message to the company. Yet, this can be dangerous, not only to your reputation of coming to conclusions prematurely but to what the company learns about your new feature. If the next day, the spike drops back down, you have to communicate all over again to try and undo the harm.
Before the product or feature launches ask and answer yourself two questions:
- “When will we know if this is successful?”
- “When will we know if this is not successful?”
Most launches inevitably have a bit of marketing surrounding them so the first day or first week is not usually reliable data. You do not want a spike and then a return to normal, you want a sustained increase. Pick a date before launch to review and share the impact of the data.
You need to account for a lot of different things when creating an experiment to test a feature or product within a company. Think through what other initiatives are going on at your company and what world events are coming up that may affect your metrics.
Common reasons your results were higher than expected:
- Marketing Campaign recently launched
- Internal usage is being factored into the data
Common reasons your results were lower than expected:
- Data isn’t being tracked correctly
- Bug in the code
- Broken links
- Weekends and National Holidays
Look at it from other people’s shoes
The unfortunate truth is that most products or features will not have a big impact, and you should be prepared for a modification or revision to have no or negative effect! People in your organization will almost always be skeptical of large positive results. You should prepare for this because the goal of your product or feature is not only to provide value to your customers but to create knowledge your company can build off of.
- What would make me skeptical of the results?
Take your answer to that question seriously and do your best to address that in your experiment design.
- Define the cohort who will be involved in your experiment, note any characteristics that make them unrepresentative
- Set a timeline for when you will evaluate the success of your experiment
- Do your due diligence when conducting an experiment to make sure your results won’t be affected by other peoples’ actions.