Measuring
Products.

A guide to product metrics, from behaviour to decisions.

Article 12 · Turn behaviour into metrics

From events to metrics

A metric is a calculation with choices about units, populations, time and comparison. Event records provide the evidence, but they do not make those choices for the team.

Event records show that selected activity occurred. A metric is the calculation and interpretation built from that evidence.

The difficult part is rarely the arithmetic. It is deciding what should be counted, which unit belongs in the denominator, who is eligible, how repeated behaviour is handled and how long an outcome is allowed to take.

A chart can calculate perfectly from the wrong choices.

Start with the unit

The same event data can be counted at several levels:

Unit Example measure What it tells you
Event record Number of form submissions How much submission activity occurred
Workflow attempt Attempts that reached first product access How often an attempt completed
User New users who reached first product access How many people completed
Account Accounts with at least one successful setup How many accounts became usable
Organisation Organisations with an approved plan How many organisations reached the outcome

These measures can all be valid. They are not interchangeable.

If one user submits a form three times, the product has three submission events, one user and possibly one workflow attempt with repeated activity. Counting events when the question is about people can make friction look like success.

State the unit of analysis before writing the formula.

Counts and unique entities answer different questions

Consider registration.form_submitted.

You might calculate:

  • total submission events
  • unique users who submitted
  • registration attempts with at least one submission
  • average submissions per attempt
  • attempts requiring more than one submission

The total event count describes volume. The unique-user count describes reach. Submissions per attempt may reveal repeated effort.

Do not call all three “form submissions” and expect readers to infer the difference.

Ratios need aligned numerators and denominators

A rate is meaningful only when the numerator could reasonably have come from the denominator.

For example:

Registration-to-access completion rate
=
new users who first accessed the product within seven days
÷
new users who started account creation

The same eligibility, cohort rule and unit should apply to both parts.

A common error is to divide completions this week by starts this week when the workflow often crosses reporting periods. Some of this week’s completions may belong to people who started earlier, while recent starters have not yet had enough time to complete.

The calculation looks current but mixes different populations.

Period and cohort metrics are not the same

A period metric describes activity inside a reporting window:

How many registration attempts completed this week?

A cohort metric follows a group that shared a starting condition:

Of the users who started registration this week,
how many reached the product within seven days?

Use period measures for volume, workload and current activity. Use cohort measures when the question concerns progression or eventual completion.

For long workflows, recent cohorts may be incomplete. Label them clearly or wait until the allowed completion window has passed.

Repeated attempts need a rule

Products often allow people to restart, retry or repeat a workflow.

Choose deliberately whether the metric uses:

  • the first eligible attempt
  • the latest attempt
  • every attempt
  • one successful attempt per user or account
  • a fixed number of attempts inside a period

Then preserve repetition as useful evidence. A person eventually completing after four failed attempts should not be indistinguishable from someone completing first time if effort and friction matter.

Possible measures include:

  • attempts per user
  • repeated-submission rate
  • first-attempt completion rate
  • eventual completion rate
  • recovery after an error

Time metrics also need boundaries

“Time to complete” is not a definition.

Specify:

  • the start event or condition
  • the end event or condition
  • whether pauses outside active use are included
  • how incomplete attempts are handled
  • the maximum observation window
  • the summary used

Durations are often skewed by a small number of long pauses. A median is usually easier to interpret than a mean, but neither is automatically correct. A distribution or percentile may show the experience more honestly.

For a workflow that people intentionally complete over several days, elapsed time may describe delay rather than effort. Consider measuring active time and elapsed time separately where the data supports it.

The metric definition carries the meaning

Metric anatomy

A metric definition is more than its formula

The arithmetic may fit on one line. The meaning depends on the choices around it.

Purpose

  1. Question What the metric should help the team understand.
  2. Decision use The monitoring, investigation, evaluation or choice it may support.
  3. Metric role Headline, diagnostic, guardrail, health or contextual measure.

Calculation

  1. Unit of analysis Event, attempt, user, account, organisation, request or another explicit entity.
  2. Numerator and denominator Aligned parts of the calculation, where a rate is used.
  3. Join and deduplication rules How records become one defensible unit without double counting.

Population and time

  1. Eligibility Who or what is allowed into the metric.
  2. Window or cohort The reporting period or follow-up rule used.
  3. Filters and exclusions Documented rules for tests, invalid records and known artefacts.

Interpretation and stewardship

  1. Source evidence and dimensions The events, systems and useful breakdowns behind the measure.
  2. Limitations and confidence What the metric hides, where evidence is incomplete and how strongly it can be used.
  3. Owner and review trigger Who maintains the definition and when it must be checked.
Two metrics with the same name can answer different questions when any of these choices differ.

The formula may occupy one line. The meaning lives in the choices around it.

A metric summarises; it also hides

A completion rate may show that something changed. It does not reveal whether the change came from fewer starts, more errors, longer delays, a different population mix, an instrumentation change or another part of the workflow.

A useful headline measure should lead back to diagnostic evidence. It should not replace it.

Define the metric before building the chart. The event data gives the calculation substance; the definition makes it trustworthy enough to use.