Choosing Metrics for Product Launch

You’re about to launch a new feature or product. You need to be able to measure its impact so that you and your company can learn from it.

This article is intended to give you a framework for prioritizing which metrics matter

1. State your default position
2. Identify your Business Drivers and Levers
3. Determine Metrics that would change your default position

State Your Default Position

What would you do if you didn’t have any metrics about the impact of your product or feature? We need to start from this position so that we know what it would take for us to change from this path

Think through what you would do about marketing, customer support, user flows, product plans without any more information than what you have right now. This is your default position, your current plan. It should include several departments and the status quo. It’s helpful to write this down on paper.

For Example:

• Marketing: spend $10,000 on paid acquisition
• Customer Support: 2 employees manning ZenDesk between 9am and 5pm EST
• Product Plan: follow the current plan to release 2 new features every month
• Engineering: continue with current headcount, producing sprints at the current pace

This plan sounds great, so what sort of information would change your mind?

Identifying Your Business Drivers and Levers:

Business Drivers = the short list of measurable properties that leads to success or failure

When we launch products, make changes, hire people, it’s all to drive the business forward. All businesses have a few key drivers of profitability.

• Cost of goods
• Revenue of goods
• Volume of goods sold
• Price of goods
• Number of paying customers

If one of these drivers change it will likely influence our default position. If marketing spend starts driving up the volume of goods sold you may invest more in it.

Business Levers = the changes you can make to influence your drivers

Some drivers are more easily changed than others. For example, we can more easily change our product’s prices than we can control the costs of goods or control the number of people willing to pay these prices.

In a business, its capacity to influence its business drivers can be viewed as a series of levers. For example, the number of paying customers is affected by sales and marketing efforts. So one way a business can affect customer number is by investing more in sales and marketing. They can pull the lever from “the current sales effort is sufficient” to “we will invest more in sales efforts”.

Ask Specific, Action-Oriented Questions

We need to map the right lever to the right driver. We must go beyond asking generic questions like “How do I get more customers?” to questions that deal in specific levers attached to a key drivers:

• “Which channels have the best conversion rate for customers?”
• “What is the ratio of earnings vs customer acquisition costs per channels so that I can adjust my strategy once this data is available?”

We want to find out what levers drive the business the most.

With generic questions like “how do I get more customers?”, there are too many levers that come to mind so their effectiveness can become confused. Do more marketing and sales, is not helpful, we need to get more specific. Questions like “which channels should my marketing budget be spent on to get most paying customers for marketing dollar spent”? Focuses us on the lever of marketing channels.

Looking at your current plan, what specific, action oriented questions do you have about it?

Indicate and enumerate them so that you have a list of specific, action oriented questions to look at. This will help us determine metrics in the next section.

Up to this point, we’ve not talked at all about metrics specifically, but you can see how this list of questions and “if I knew” statements can guide us to picking the right metrics.

Metrics That Would Change My Default Position

Let’s focus in on the default position, the question, and the levers columns. Are there metrics that clearly answer the question and would change your position by telling you which direction to pull a lever?

Once again, these metrics allow the business to make decisions about when and how to pull certain levers. And not all metrics are simple to construct. For example, there could be much more debate over what counts as “fixing issues” vs planning, or what conversions matter. But if you put in the work, you will find that your decision-making, company alignment, and outcomes will all improve over time.

Summary

In conclusion, companies and teams that put in the time to visualize potential outcomes and default actions, determine what matters to the business, and decide what can be changed under which circumstances will reap significant benefits.

There is no shortage of important decisions for product companies approaching a launch. And with every decision, it can feel like the stakes could not be any higher, but whenever possible remind yourself that your company’s success has more to do with the overall quality of data-driven decisions made than any single analysis. Whether your next big data-driven decision comes in a quarter or in a week, take care to improve your process and appreciate the clarity, growth, and excitement that comes from a decision well made.

• State your default position - What would you do without data
• Identify your levers - What could you change if you did have data
• Determine metrics that would change your position - How much would that metric need to change in order to pull a lever

Written by: Lu Wang
Reviewed by: Matt David