Choosing Metrics for Product Launch

Last modified: August 27, 2019 • Reading Time: 5 minutes

You’re about to launch a new feature or product. You need to be able to measure its impact so that you and your company can learn from it.

This article is intended to give you a framework for prioritizing which metrics matter

  1. State your default position
  2. Identify your Business Drivers and Levers
  3. Determine Metrics that would change your default position

State Your Default Position

What would you do if you didn’t have any metrics about the impact of your product or feature? We need to start from this position so that we know what it would take for us to change from this path

Think through what you would do about marketing, customer support, user flows, product plans without any more information than what you have right now. This is your default position, your current plan. It should include several departments and the status quo. It’s helpful to write this down on paper.

For Example:

  • Marketing: spend $10,000 on paid acquisition
  • Customer Support: 2 employees manning ZenDesk between 9am and 5pm EST
  • Product Plan: follow the current plan to release 2 new features every month
  • Engineering: continue with current headcount, producing sprints at the current pace

This plan sounds great, so what sort of information would change your mind?

Identifying Your Business Drivers and Levers:

Business Drivers = the short list of measurable properties that leads to success or failure

When we launch products, make changes, hire people, it’s all to drive the business forward. All businesses have a few key drivers of profitability.

  • Cost of goods
  • Revenue of goods
  • Volume of goods sold
  • Price of goods
  • Number of paying customers

If one of these drivers change it will likely influence our default position. If marketing spend starts driving up the volume of goods sold you may invest more in it.

Business Levers = the changes you can make to influence your drivers

Some drivers are more easily changed than others. For example, we can more easily change our product’s prices than we can control the costs of goods or control the number of people willing to pay these prices.

In a business, its capacity to influence its business drivers can be viewed as a series of levers. For example, the number of paying customers is affected by sales and marketing efforts. So one way a business can affect customer number is by investing more in sales and marketing. They can pull the lever from “the current sales effort is sufficient” to “we will invest more in sales efforts”.

Department Drivers This Department Affects Levers This Department Owns
Marketing Perceived value of productVolume of new leads Marketing budget and channelsBranding
Sales Number of paying customers Sales team staffingSales strategy
Product Price of goods Product roadmap Sprint planning
Engineering Cost of goods Sprint planning

Ask Specific, Action-Oriented Questions

We need to map the right lever to the right driver. We must go beyond asking generic questions like “How do I get more customers?” to questions that deal in specific levers attached to a key drivers:

  • “Which channels have the best conversion rate for customers?”
  • “What is the ratio of earnings vs customer acquisition costs per channels so that I can adjust my strategy once this data is available?”

We want to find out what levers drive the business the most.

With generic questions like “how do I get more customers?”, there are too many levers that come to mind so their effectiveness can become confused. Do more marketing and sales, is not helpful, we need to get more specific. Questions like “which channels should my marketing budget be spent on to get most paying customers for marketing dollar spent”? Focuses us on the lever of marketing channels.

Question What business driver does this address? If I knew the answer, what lever could I pull? Is this a good question?
How do I get more customers? Volume of paying customers It depends on the answer… maybe I can act on the answer, maybe not. Too vague…
Which marketing channels have the best conversion rate for customers? Volume of paying customers I could spend my marketing dollars differently Much better
What is the ratio of customer revenue vs customer acquisition costs per channels? Unit economics I could invest in more profitable marketing channels or find ways to adjust my prices Much better!

Looking at your current plan, what specific, action oriented questions do you have about it?

Indicate and enumerate them so that you have a list of specific, action oriented questions to look at. This will help us determine metrics in the next section.

Department Default Position Specific, Action-Oriented Question If I knew, I would change my default position by pulling which levers?
Marketing Spend $10,000 on paid acquisition Which channels have the best conversion rate? I could increase my budget on the best channels
Customer Support 2 employees manning ZenDesk from 9am - 5pm EST How many tickets get missed because there wasn’t a 3rd person available? I could hire another employee
    How many tickets get missed because we don’t have more hours of coverage? I could hire a support service
Product Follow the current plan to release 2 new features every month How many features were released with bugs that forced them back into development? I could adjust sprint planning to accommodate for more planning and iteration time.

Up to this point, we’ve not talked at all about metrics specifically, but you can see how this list of questions and “if I knew” statements can guide us to picking the right metrics.

Metrics That Would Change My Default Position

Let’s focus in on the default position, the question, and the levers columns. Are there metrics that clearly answer the question and would change your position by telling you which direction to pull a lever?

Default Position Specific, Action-Oriented Question If I knew, I would change my default position by pulling which levers? Metrics
Spend $10,000 on paid acquisition Which channels have the best conversion rate? I could increase my budget on the best channels Conversion rate per channel
2 employees manning ZenDesk from 9am - 5pm EST How many tickets get missed because there wasn’t a 3rd person available? I could hire another employee Missed tickets and ticket wait time between 9am and 5pm
  How many tickets get missed because we don’t have more hours of coverage? I could hire a support service Number of tickets received outside of business hours
Follow the current plan to release 2 new features every month How many features were released with bugs that forced them back into development? I could adjust sprint planning to accommodate for more planning and iteration time. Ratio of work units spent on feature planning vs development vs fixing issues over time

Once again, these metrics allow the business to make decisions about when and how to pull certain levers. And not all metrics are simple to construct. For example, there could be much more debate over what counts as “fixing issues” vs planning, or what conversions matter. But if you put in the work, you will find that your decision-making, company alignment, and outcomes will all improve over time.

Summary

In conclusion, companies and teams that put in the time to visualize potential outcomes and default actions, determine what matters to the business, and decide what can be changed under which circumstances will reap significant benefits.

There is no shortage of important decisions for product companies approaching a launch. And with every decision, it can feel like the stakes could not be any higher, but whenever possible remind yourself that your company’s success has more to do with the overall quality of data-driven decisions made than any single analysis. Whether your next big data-driven decision comes in a quarter or in a week, take care to improve your process and appreciate the clarity, growth, and excitement that comes from a decision well made.

  • State your default position - What would you do without data
  • Identify your levers - What could you change if you did have data
  • Determine metrics that would change your position - How much would that metric need to change in order to pull a lever

Written by: Lu Wang
Reviewed by: Matt David

Next – Types of Data Analysis

Get new data chapters sent right to your Inbox