Measurement review checklist

A measurement review checks whether a defined slice of the measurement system still supports a real decision with appropriate confidence.

Review one important workflow, metric, dashboard or change at a time. A broad review of “all analytics” usually produces vague findings and no owner.

Record the scope first

Review scope:
[Workflow, metric, dashboard, release or decision being reviewed]

Reason for review:
[Product change / release / confidence issue / new use / ownership change / planned retirement]

Decision or use:
[What the measurement should help the team do]

Review participants:
[Accountable owner, definition steward, implementation owner and relevant contributors]

Evidence period:
[Dates, cohorts, versions and environments examined]

A review without a declared scope can make healthy components look weak because unrelated problems are mixed together. It can also make a fragile metric look healthy because only the dashboard presentation is inspected.

1. Purpose and decision

• Is the product question still clear and worth answering?
• Is there a real monitoring, diagnostic, evaluation, prioritisation or maintenance use?
• Is the metric or dashboard still used in that decision?
• Has the audience, target, policy or operating context changed?
• Would anything meaningful be lost if this measurement disappeared?

2. Workflow, unit and boundary

• Is the workflow definition current?
• Is the unit of analysis explicit?
• Are actors and object relationships clear?
• Are entry, eligibility and completion conditions still defensible?
• Are the observation window and cohort-maturity rule appropriate?
• Are repeated attempts, optional routes, pauses, exits and reversals handled?
• Is relevant off-platform or operational activity acknowledged?
• Is digital completion kept separate from downstream value where necessary?

3. Events and source evidence

• Can each important event be traced to an observable fact or persisted state?
• Is the authoritative source or producer known?
• Are fire and non-fire conditions documented?
• Are retries, duplicates, replay, late arrival and reversals handled?
• Are required identifiers and properties present and valid?
• Can event records be joined to the intended unit without inflating counts?
• Are source completeness, latency and reconciliation understood?
• Does the evidence still represent the current product and state model?
• Is qualitative or operational evidence needed to explain what events cannot show?

4. Metric definition and interpretation

• Is the metric defined in the catalogue rather than only inside a chart or query?
• Do the numerator and denominator use the same intended unit?
• Is the eligible population or cohort explicit?
• Are formulas, joins, filters, exclusions and deduplication rules visible?
• Are reporting periods distinguished from observation windows?
• Are dimensions stable, purposeful and safe to interpret at the available sample size?
• Are provisional or incomplete cohorts labelled?
• Are supporting diagnostic, guardrail and outcome measures available?
• Are limitations and alternative explanations visible?
• Are causal claims stronger than the evidence supports?

5. Dashboard, report or analysis use

• Does the view have one declared job: monitor, diagnose, evaluate or assess measurement health?
• Does it show the context required for that job?
• Are comparisons, baselines, versions and caveats visible?
• Are empty, delayed, incomplete and broken-data states handled?
• Can a reader tell what action or investigation may follow?
• Are unused charts, duplicate metrics or expired evaluation views still present?
• Does the view direct people to the canonical definitions?

6. Privacy, access and proportionality

• Does every identifier and property have a defined measurement purpose?
• Is personal, sensitive or free-text data excluded unless genuinely required and authorised?
• Are access restrictions appropriate?
• Are retention and deletion expectations documented?
• Have new uses changed the privacy, consent or governance position?
• Could a less identifying or more aggregated measure answer the question?
• Are small groups protected from unsafe or misleading disclosure?

7. Ownership and operating health

• Is someone accountable for the question and decision use?
• Is there a steward for workflow, event and metric definitions?
• Is implementation reliability owned?
• Are changes reviewed before release where measurement matters?
• Are failures escalated and visible to affected users of the data?
• Are review triggers defined?
• Can ownership be handed over without relying on one person’s memory?
• Is retirement possible, including downstream consumers and documentation?

Classify what you find

Not every finding is measurement debt.

This distinction prevents a broken production metric from waiting in a backlog and prevents every desirable improvement from being labelled debt.

Produce an action record

Finding:
[What was observed]

Classification:
[Incident / gap / debt / accepted limitation / healthy]

Decision impact:
[Which decision, workflow or audience is affected]

Action:
[Fix / design / consolidate / caveat / retire / explicitly accept]

Owner:
[Accountable person, role or team]

Due or review trigger:
[Date or event that causes the action to be revisited]

Validation:
[How the team will know the action worked]

Communication:
[Who needs to know about the issue, change or limitation]

A review that produces twenty unowned observations has not improved the system. Prefer a small number of actions with clear decision impact, ownership and validation.

Example review finding

Finding:
The request acceptance dashboard counts quote.accepted events rather than distinct eligible requests, and recent requests are included before the 30-day window closes.

Classification:
Live incident for the dashboard; definition debt in the underlying metric query.

Decision impact:
Marketplace performance appears lower for recent cohorts and can be inflated by duplicate acceptance delivery.

Action:
Replace the dashboard calculation with the catalogue definition, deduplicate by acceptance_change_id, count distinct request_id and label immature cohorts as provisional.

Owner:
Analytics implementation owner, reviewed by the metric steward and marketplace product lead.

Validation:
Reconcile the result against persisted accepted request states and confirm stable cohort behaviour across two reporting cycles.

Communication:
Add a dashboard notice and inform regular review participants before the corrected series is used.

The checklist is successful when the team can say what remains trustworthy, what needs action, who owns that action and which future change should trigger another review.