How to measure feature success: connect the outcome to a behaviour, a metric and a guardrail

A feature that ships and gets used has not yet succeeded; it has only been adopted. Success means a behaviour changed in a way that moves an outcome the business cares about, without breaking something else. Kerry Rodden, Hilary Hutchinson and Xin Fu, writing up the metrics process they applied to more than 20 products and projects at Google (CHI 2010), put the rule in one line: "No matter how user-centered a metric is, it is unlikely to be useful in practice unless it explicitly relates to a goal, and can be used to track progress towards that goal."

The procedure below connects one stakeholder request to that goal through four more links, then runs it twice on original examples: a B2B helpdesk feature and a consumer meal-planning feature. It ends with a one-page plan you can attach to any feature ticket.

Why is "the feature shipped and people use it" not success?

Usage numbers answer a different question. A new button gets clicked because it is new, because it replaced the old path, or because a launch email pointed at it. None of those means the user now does something different that matters to the business. A team that reports "4,000 people used suggested replies in week one" has measured curiosity.

The chain that turns a feature into something measurable has five links:

Link	The question it answers	Failure if skipped
Business outcome	Which number the company already tracks should move?	The feature has no reason to exist that anyone can check
User behaviour	Who must do what differently, how often, instead of what?	You measure clicks on the feature instead of the change it was meant to cause
Metric	Which normalised number shows that behaviour, with a baseline, target and date?	Raw counts rise with user growth and look like success
Instrumentation	Which events and properties compute the metric and its comparison?	The metric cannot be calculated on launch day
Guardrail	What must not get worse, and by how much is too much?	A win that moves cost somewhere else is reported as a win

The behaviour link is the one most plans skip, and it is the one that makes the rest possible. Rodden and colleagues call this step choosing signals, and their advice is to pick signals that are "sensitive and specific to the goal", which in practice means a behaviour the feature can plausibly change and little else does.

How do you walk a feature request from outcome to guardrail?

Start from the request, write the problem underneath it (the problem statement template covers that step), then work through the links in order:

1. Name the business outcome. Pick one number the company already reports: renewal rate, expansion revenue, support cost per account, retention. If the feature cannot be connected to any of them in one sentence, ask why it is on the roadmap.
2. Write the behaviour change as a sentence. Use the shape "[who] will [do what] [how often] instead of [current behaviour]." The "instead of" clause is the test: if you cannot say what users do today, you do not yet know what the feature replaces.
3. Turn the behaviour into a metric. Use a ratio or a rate per user or account, not a total. Rodden and colleagues are explicit that "Raw counts will go up as your user base grows, and need to be normalized; ratios, percentages, or averages per user are often more useful." Record the baseline from the last four to eight weeks, the target, and the date you will read it.
4. Specify the events and properties. List the events that compute the metric and its denominator, plus one adoption event for diagnosis. Name them as object plus action and keep detail in properties: the Twilio Segment tracking plan guidance puts it as "events should be generic and high level, whereas properties are specific and detailed."
5. Choose one or two guardrails with a margin. Spotify's experimentation team defines guardrails in its write-up on multi-metric decisions as "Metrics that we don't expect to improve, but we want evidence that they're not deteriorating by more than a certain margin." The margin is the part to write down now, while nobody knows the result.
6. Decide the comparison. Name the group the feature's users will be compared with: an A/B split, a holdout, or accounts that get it later. Without a comparison, the read date produces a before-and-after chart and an argument.

Each step should take minutes, not days. If one takes days, that link is where the real uncertainty about the feature sits, and finding it before build is the point of the exercise.

What does the chain look like for a B2B feature?

The request, from the Head of Sales at a fictional helpdesk SaaS: "Add AI suggested replies; two prospects asked for it." The problem underneath, after a week of looking at support data and talking to three customer admins: agents at mid-size accounts start every first reply from a blank box, so first response times slip on Monday mornings, and accounts with contractual response targets complain at renewal.

Link	Helpdesk example (fictional numbers)
Business outcome	Gross renewal rate of mid-size accounts (20 to 200 agents), currently 86 percent a year
Behaviour	Agents at those accounts will send the first reply to a new ticket by editing a suggested draft, instead of writing from a blank box or a generic macro
Success metric	Median first response time on new tickets at mid-size accounts: baseline 4 hours 10 minutes, target 3 hours, read 8 weeks after full rollout
Adoption diagnostic	Share of first replies sent from a suggestion (no target; explains the success metric)
Events and properties	Ticket Created (account_id, channel, priority); Suggestion Shown (ticket_id, model_version); Reply Sent (ticket_id, account_id, is_first_reply, source: suggestion, macro or blank, edit_ratio)
Guardrails	Ticket reopen rate within 7 days, baseline 8 percent, must not exceed 9 percent; customer satisfaction on first-reply tickets, baseline 4.3 out of 5, must not fall below 4.2
Comparison	Staged rollout: half of mid-size accounts get the feature in month 1, the rest in month 3

Three choices in that table carry the learning. The success metric is first response time, not suggestion usage, because faster first replies are the behaviour the renewal risk depends on. The source property on Reply Sent lets the team compare suggestion-based replies with blank and macro replies inside the same account, which is the only way to tell whether suggestions speed agents up or only get used by agents who were already fast. And the reopen guardrail exists because the cheapest way to cut response time is a fast, wrong answer; a feature that wins on speed by pushing tickets back into the queue has moved cost, not removed it.

Renewal is the outcome but not the read-date metric. It moves once a year and depends on price, competitors and the account's own budget. The plan names it so everyone knows why response time matters, then judges the feature on the behaviour metric that can move inside a quarter.

What does the chain look like for a consumer feature?

The request, from the CEO of a fictional meal-planning subscription app: "Build a shopping list." The problem underneath: subscribers who pick recipes for the week still have to copy ingredients into a notes app, and many stop planning after their first month.

Link	Meal-planning example (fictional numbers)
Business outcome	Month-2 subscription renewal rate, currently 61 percent
Behaviour	Subscribers will plan at least three meals for the week and shop from one generated list, instead of choosing recipes on the day
Success metric	Share of active subscribers with a "planned week" (3 or more recipes in the plan and a list generated from it): baseline 18 percent, target 25 percent, read after 6 weekly cycles
Adoption diagnostic	Share of subscribers who generate any list
Events and properties	Meal Plan Updated (week_start, recipes_count); Shopping List Generated (source: plan or single recipe, items_count); List Item Checked (list_id); Cook Mode Started (recipe_id, from_plan)
Guardrails	Cook Mode sessions per active subscriber per week, baseline 2.4, must not fall more than 5 percent; list screen crash rate under 0.5 percent
Comparison	A/B test on new subscribers for the first 6 weeks, then a 5 percent holdout for one renewal cycle

The cooking guardrail is the non-obvious one. A shopping list can make planning feel like progress while subscribers cook less, for example if the list nudges them toward bulk shopping and fewer, larger meals. Renewal in a meal-planning app depends on cooking, not on lists, so the plan protects the behaviour the business actually runs on. The from_plan property on Cook Mode Started makes it possible to see whether planned meals get cooked, the link between the success metric and the renewal outcome.

The comparison has two stages for a reason. Six weeks of A/B data answers whether planned weeks go up. Only the holdout, run through a full renewal, answers whether planned weeks lead to renewals, and that second answer is what decides whether the next quarter goes into planning features.

Which signals look like feature success but are not?

Run every launch readout against these before calling it a win. Each has one question that exposes it.

Signal that looks like success	Why it misleads	The question that exposes it
Feature usage grew every week	Raw counts grow with the user base and with launch promotion	Did usage grow as a share of eligible users, after the launch email effect faded?
Users of the feature retain better	Engaged users adopt new features first; the feature may not have caused anything	How do adopters compare with a randomised or holdout group, not with non-adopters?
Strong first two weeks	New things get tried because they are new	Is the rate in weeks 5 to 8 still above baseline?
Old path usage fell as new path rose	Users moved from one path to another without doing more of the behaviour	Did the total behaviour (all paths together) change, or only its route?
Success metric improved	Something else may have got worse to pay for it	What did the guardrails do, against the margin written before launch?
Leading metric moved, outcome did not	The behaviour may not be the lever you assumed	After one full outcome cycle, did the outcome move in the group that got the feature?

The second row deserves special care because it survives most reviews. Comparing feature adopters with everyone else almost always shows the adopters doing better, and the comparison says nothing about the feature. When a randomised comparison is impossible, pre/post vs difference in differences vs A/B test covers the fallbacks and what each can support. When an A/B test is possible, how to run an A/B test covers sizing it.

How is this different from choosing a North Star or an activation metric?

A North Star is a single company-level number that represents delivered value, and the North Star metric examples guide covers how companies choose one. An activation metric marks a new user's first moment of value, covered in how to define an activation metric. A feature measurement plan sits below both: it says which behaviour one feature should change and how that behaviour connects upward, usually to a branch of the company's metrics tree.

The practical difference is the read date. A North Star is tracked continuously; a feature plan has a date by which the feature is judged, and a decision attached to each possible result. That is what makes it a decision tool rather than a dashboard.

What goes on a one-page measurement plan?

Attach this to the feature ticket before build starts. If a field cannot be filled, mark it "unknown" and treat it as a risk, not as a gap to fill after launch.

FEATURE MEASUREMENT PLAN
Feature:                         Owner (PM):        Analyst:
Request (as received, who, date):
Problem underneath (one sentence):

1. Business outcome (existing company metric + current value):

2. Behaviour change
   [Who] will [do what] [how often] instead of [current behaviour]:

3. Success metric
   Definition (numerator / denominator):
   Baseline (period):        Target:        Read date:

4. Adoption diagnostic (no target):

5. Events and properties (object + action; details as properties)
   Event:                     Properties:
   Event:                     Properties:
   Owner of instrumentation:  Verified in staging on (date):

6. Guardrails (max 2)
   Metric:                    Baseline:      Must not go past:
   Metric:                    Baseline:      Must not go past:

7. Comparison group (A/B, holdout, staged rollout):

8. Decision rule
   Success metric hits target, guardrails hold  -> keep, invest further
   Success metric short of target, guardrails hold -> keep or simplify; record why
   Any guardrail past its margin                 -> rework or roll back
   Behaviour unchanged after read date           -> retire or rethink the problem

9. Known traps for this feature (novelty, path shift, adopter bias, other):

The decision rule in section 8 is the part that keeps the plan honest. Written before launch, it turns the read-date meeting into checking which row applies instead of debating what success meant, the same discipline the product decision-making framework applies to calls without data.

Is a measurement plan too much process for a small feature?

For a one-day copy change, yes, the full page is overkill, and pretending otherwise teaches teams to fill in forms without thinking. The honest scale: a small, reversible change needs the behaviour sentence and one guardrail, written in the ticket. A feature that takes a quarter of engineering time, touches pricing or renewal, or ships to every customer at once needs the whole page.

The cost of skipping it shows up later. A feature without a behaviour sentence cannot fail, because nobody said what it should change, and features that cannot fail are never removed. Ten minutes on the plan before build is how a team keeps the option of saying "this did not work" on the read date, and of reading a result that is real but too small to matter, the case statistical vs practical significance works through.

Where can you practise defining success before solving?

Builders Camp runs Product Sense as a 2-week bootcamp with 3 live sessions of 90 minutes, also available self-paced. Its published topics include problem framing (separating requests from real problems and defining success before choosing a solution) and data-informed intuition (working with weak signals, incomplete data and misleading metrics).

See the Product Sense bootcamp

Try it on the feature your team is building now: write only the behaviour sentence, with its "instead of" clause, and show it to the engineer building the feature. If the two of you describe the current behaviour differently, you have found the first thing to measure.