How do you choose what to measure?

How do you choose what to measure?


5 minute read

Choosing product KPI

Choosing what you need to measure can be simple, but it can also be very tough. Ensuring your chosen metrics are in line with your product strategy is key, but how can we make sure we are tracking the right things? By focusing on the metrics that matter the most, and considering what really drives value for your product you can identify KPI that will help you steer your way to success.

Avoiding vanity metrics

Look how many users I’ve got, I’m doing great!

The Lean Startup is an excellent book by Eric Ries. It highlights the pitfalls of the “Vanity Metric”. Something which you can track and measure, but whilst it feels good, it’s not truly measuring your success against your strategy, and it’s not an indicator that your product is driving value for your users.

Some good examples of vanity metrics could be:

  • Number of sessions in a web application - just because you have more sessions, doesn’t mean your users are valuing your service more. Perhaps they’re being caught by an error and having to log in. Perhaps they are struggling to find what they want in one go, leaving them frustratingly accessing your service several times to achieve their goal.
  • Number of installs of a mobile app - installing an app is obviously good, and a step in the right direction. But it’s not a valid KPI in its own right. By simply installing the app, the user has obtained little or no value.

So to avoid vanity metrics, make sure the KPI you’re tracking are meaningful contributors towards your strategy.

Measuring in a changing environment

Another challenge for picking sensible metrics is that nothing remains the same for long. It’s likely that you’re adding new users through a successful sales & marketing function. So again, if your raw number of users using your product goes up month on month, then that’s good - but the success of user aquisition could be masking other problems. For example, in a month the net position may be positive, but user retention could also be a masked problem.

$$ netUsers = newUsers - lostUsers $$

In an environment when users are being added, tracking raw figures for feature usage or other goal completions only tells part of the story. You’ll want to focus on the proportion of users who complete particular goals, relative to the total users that month.

$$ goal = \frac{usersCompleting} { totalUsers } * 100 $$

When looked at as a percentage (using the oh so complex formula above) then you can see a meaningful metric.

Defining sensible cohorts of users

The final thing we’ll look at today is what subset of our user base we’re measuring.

For some KPI it can make sense to look at and track different cohorts of users separately. For example, tracking the goal completion of “new” users (e.g. users who have activated in the past month) separately from the general user base. This can help identify changes in your on-boarding or acquisition processes - for better or worse.

Looking at different cohorts individually can be very helpful to make KPI more meaningful. Some typical cohorts might be:

  • Age of account
  • Referral source / account type
  • Frequency of use
  • Device type (e.g. mobile, table, PC…)

Of course, depending on the nature of your service there are bound to be others. With many of these things, you don’t know if the cohort will be statistically interesting or different from the general user population until you look!

One metric to rule them all

We can’t all be as focused on a single outcome as the Dark Lord Sauron, and he probably isn’t a good role model for successful product management. It’s unrealistic to think that there is one KPI that you can track that will give you enough insight in isolation. However, like Sauron, having an ever watchful eye across the land is probably the best approach. Track a range of KPI, absolute number and ratios, cohorts and the whole user base, and you’ve got a good chance of being able to see the bigger picture as part of an evolving and changing product environment.

So, track like a pirate - the AARRR model

One helpful model to stimulate key metrics is the Pirate Model by Dave McClure - check out the original slide deck that outlines the model.

In summary, have metrics for each of the following:

  • Aquisition - gaining new users
  • Activation - users signing up or activating their account, obtaining their first value
  • Retention - users continue to use the service
  • Referral - delighted users spread the word
  • Revenue - users create value for your company

Tracking a series of metrics like this lends itself to a funnel - you’ll always have more users at the top than the bottom. Metabase is a great way of easily creating funnel visualisations.

So, I hope that helps stimulate some KPI ideas for you! If you have any special tips or KPI that work for you, then please share them in the comments! By sharing examples of metrics that work, and those that don’t, we can all learn and be more successful.

comments powered by Disqus