Mindful product metrics

Be mindful about the product metrics you track to ensure you understand what they are telling you


4 minute read

Mindful product metrics

As a new feature for the site, I’ve started building a list of common product KPI’s and metrics with simple definitions. It’s certainly a work in progress, and already feels like one of those tasks that’ll never quite be complete, but hey - I thought it might be helpful. If there are obvious KPIs missing (I’m sure there are!) then let me know in the comments.

A recent addition has been the definition for average session length. This is an interesting metric to track, but like with so many high level metrics care is needed.

Let’s see why.

Imagine you’re a product manager, looking at some usage stats for the year. You’re looking at the average session duration for users - the average amount of time a user spends on your product per visit. Calculating the average session duration is fairly simple: for each session look at the first and last action the user performed - that’s the session duration. Now, take an average of the recorded session durations for the month and plot a chart and let’s see what trends we can see.

Oh great, my average session length is increasing

Well, that’s great. Or at least, it could be great.

An increase in average session time could be an indicator of a user base that’s loving your product. They clearly seem to enjoy it if they’re using it and they’re using it for increasingly long periods of time.

However, a graph like this could also mean bad news. Perhaps that change you made to the checkout process is confusing people - they’re having to spend longer in your product to achieve the goals they were previously flying through. An increase in average session length could be an indicator of problems with the user experience of the product.

Woop! My average session length is going down

Well, the same is true here. It could be good - you’ve nailed the user experience and users can get the job done in half the time. But it could also be bad - usage is becoming more transient and superficial, users aren’t engaging with the product for as long as they used to.

It’s important to consider any external factors that might be influencing the metric. Have any changes been released? Has there been a significant number of new users? To find out if the trends you see are good or bad, we need more information.

So, is it worth tracking then?

It is worth tracking as long as you’re mindful about what the metric is showing. It’s a classic example of needing to track a range of metrics to be sure of what is really going on. Only by combining the average session length with a number of other metrics could you create a robust hypothesis on which to take any further action.

Like a good meal and fine wines, there are some pairings of metrics that just go well together. When looking at a broad metric like average session length, its perhaps good to also try and look at more goal-based metrics. For example, the average length of time taken by users to complete an order. It’s also useful to track the proportion of users completing specific goals.

Being mindful about your metrics

With goal or activity based metrics, you can make more informed conclusions about the higher level KPIs. It’s only by being mindful of what the metric indicates that you can then move past simply reacting to a number going up or down. By looking past the raw number, combining it with other metrics and thinking about the bigger picture you can start to have a higher awareness of the changes your seeing.

In the Lean Startup Ries tells us to beware vanity metrics - things we track because they superficially feel good. Metrics like average session length could feel like a vanity metric, but it does tell us something helpful. The problem is that what it is telling us isn’t clear without other metrics and more thought.

What are you tracking? Have you had any examples where the numbers aren’t quite telling the full story? Let me know in the comments.

comments powered by Disqus