Experiment backlog template for product teams, with a learning log that compounds

Most A/B test ideas lose, and a team that does not write down its losses will keep testing them. Ron Kohavi and colleagues reported in Online Experimentation at Microsoft that "only about 1/3 of ideas improve the metrics they were designed to improve", with roughly another third flat and a third negative. The same paper names the fix: "One of our goals at ExP is to share results widely and enable the 'institutional memory' about what worked and what did not so groups at Microsoft can stand on the shoulder of others instead of stepping on their toes."

A backlog and a learning log are that institutional memory at team scale. The template below has both, filled with six entries each for one fictional product: a language-learning app with a free tier and a paid plan, where every number is invented for the example but computed with the standard formulas so you can check the arithmetic.

Why does an experiment backlog need more than a list of ideas?

An idea list ranks ideas by how much people like them. An experiment backlog ranks them by what a testing slot is worth, and that needs numbers the idea list never asks for: how much traffic the surface gets, how big an effect you need to see, and how long the test will hold that surface hostage.

The estimated sample field does most of the work. An idea that needs 40 weeks of traffic to detect the effect worth shipping is not a test; it is either a bolder change waiting to be designed or a judgment call to make without a test. Finding that out when the idea enters the backlog costs ten minutes with a sample size calculator. Finding it out after launch costs the slot.

What fields go in the experiment backlog template?

Each idea gets one row with these fields. The first five come from the test plan described in how to run an A/B test; the scoring fields are what turn a list of plans into a queue.

Field	What to write	Why it is there
Hypothesis	One change, the audience, the expected behaviour and the reason from evidence	A loss teaches nothing without a stated reason
Primary KPI and baseline	The single metric that decides the test, with its current rate	Needed for the sample size
Guardrails	Two or three metrics that must not get worse, with thresholds	Catches a win that moves cost elsewhere
Minimum detectable effect (MDE)	The smallest lift worth shipping, in points	Sets the sample; also the bar for practical significance
Estimated sample and run time	Users per variant, then traffic days plus the outcome window, rounded up to whole weeks	Shows the true cost of the slot
Prior win probability	The win rate of past tests on this surface, from the learning log	Replaces the proposer's confidence
Value if it wins	Yearly value of the MDE: eligible users per day, times 365, times the MDE, times the value of one extra conversion	Makes ideas on different surfaces comparable
Score	Prior times value, divided by weeks of slot	The queue order

A plain-text row to paste into whatever table your team uses:

ID:
Hypothesis (change / who / behaviour / because):
Surface:
Primary KPI + baseline:
Guardrails + thresholds:
MDE (points):
Users per variant:          Traffic days:       Outcome window (days):
Run time (whole weeks):
Prior win probability (source: learning log rows ...):
Value per extra conversion (EUR):
Value if it wins, per year (EUR):
Score = prior x value / weeks:
Status: idea / ready / running / in log

The outcome window is the field teams forget. A test on day-7 retention cannot be read until the last user enrolled has had seven days, so it holds the slot a week longer than its traffic suggests. A trial-length test holds it for the length of the longer trial.

How do you score and prioritise the experiments in the backlog?

Score each idea as expected value per week of slot: prior win probability times the yearly value of a win at the MDE, divided by the weeks the test occupies. The score is crude on purpose. It forces three conversations that usually stay implicit: how often ideas like this have actually won, what a win is worth in money, and what the test displaces.

The formula is close to the ICE scoring framework with each letter replaced by a number you can check: impact becomes value at the MDE, confidence becomes the observed win rate, and ease becomes weeks of slot. Rank within each surface, because two tests on different surfaces with separate users can run at the same time, while two tests on the same paywall compete for the same users.

What does a filled experiment backlog look like?

Six ideas for the fictional language-learning app. Sample sizes use 80 percent power and 5 percent two-sided significance, with traffic split evenly between two variants.

ID	Hypothesis (short)	Primary KPI, baseline	MDE	Users per variant	Run time	Prior	Value if it wins, per year	Score per week
B1	Show the annual plan as a monthly price first on the paywall, because interviews show users compare it with the monthly plan	Paywall to purchase, 4.0 percent	0.5 points	25,551	2 weeks (6,000 paywall users a day)	0.30	€438,000	€65,700
B5	Extend the free trial from 7 to 14 days, because most trial users who convert do so after their tenth lesson	Trial to paid, 38 percent	2 points	9,336	5 weeks (900 trial starts a day, plus 14-day window)	0.22	€262,800	€11,563
B2	Ask new users for a daily minutes goal in onboarding, because users who set a goal in research sessions came back more	Day-7 retention, 28 percent	1.5 points	14,291	3 weeks (4,000 signups a day, plus 7-day window)	0.20	€131,400	€8,760
B3	Offer a streak repair the day after a missed lesson, because support tickets show users quit after losing a long streak	Return within 48 hours after a break, 35 percent	3 points	4,042	2 weeks (1,500 eligible a day, plus 2-day window)	0.22	€65,700	€7,227
B4	Send the daily reminder at each user's usual lesson time instead of 7 PM	Lesson started within 2 hours, 9 percent	0.6 points	36,780	1 week (20,000 users a day)	0.15	€13,140	€1,971
B6	Remove the skip button from the placement test	Placement test completion, 62 percent	2 points	9,147	2 weeks (2,500 a day)	0.20	€18,250	€1,825

The values assume €40 per extra paid conversion, €6 per extra retained user, €4 per returning user, €0.30 per extra lesson started and €1 per completed placement test, all made up for the example.

Three things the table shows that an idea list would hide. B1 dwarfs everything because the paywall is where money changes hands and the traffic is large, so it gets the paywall slot first. B5 has the second-best score but occupies its surface for five weeks; it can run in parallel with B1 only if trial starts and paywall views are separate populations, which in this app they are not, so it queues behind B1. B6 scores so low that testing it is the wrong call: it is cheap and reversible, so the team ships it behind a flag and watches onboarding completion, or drops it.

What goes in the learning log?

The learning log is one row per finished test, written within a week of the decision while the people who ran it still remember why. These fields make the row useful a year later to someone who was not there.

Field	What to write
What was tested	The hypothesis as written before launch, unedited
Decision row	Which row of the pre-launch decision rule applied: ship, rework, stop, inconclusive, or stopped for harm
Effect with interval	Point estimate and 95 percent confidence interval in points, plus relative lift
Guardrails	Each guardrail's movement and whether it breached
Pre-registered segments	The two or three segments named before launch and their effects; nothing found after the fact
What the losing variant taught	The assumption the result broke, and the next hypothesis it suggests
Flags	Early stop, sample ratio problem, novelty period, tracking change: anything that should lower trust in the number

The decision row matters more than it looks. It records the call against the rule that existed before the data, so a reviewer can see whether the team followed its own rules. The interval matters because the point estimate alone overstates what you know; statistical vs practical significance shows how the low end of an interval can turn a clean win into a loss.

What does a filled learning log look like?

Six finished tests from the same fictional app, from earlier quarters. All intervals are 95 percent, computed from the counts.

ID	What was tested	Decision row	Primary effect (95% CI)	Guardrails	Pre-registered segments	What the loser taught
L1	Annual plan preselected on paywall (26,000 per variant)	Ship	4.10 to 4.70 percent, +0.60 points (+0.25 to +0.95), +14.6 percent relative	Refund requests flat	iOS +0.7, Android +0.5 points; both positive	Not a loss; the next question is how the annual plan is framed (B1)
L2	Let users take a first lesson before creating an account (14,500 per variant)	Inconclusive	27.6 to 28.0 percent day-7 retention, +0.40 points (minus 0.63 to +1.43)	Account creation rate down 2 points, within threshold	New users from ads vs organic: no difference	Onboarding order changes move day-7 retention by about a point at most; a 3-point MDE there is fantasy
L3	Second daily reminder at 9 PM (38,000 per variant)	Rework: guardrail breach	Lesson within 2 hours 9.0 to 9.9 percent, +0.90 points (+0.48 to +1.32)	Notification opt-out 2.0 to 2.6 percent, +0.60 points (+0.39 to +0.81), breached	Heavy vs light users: opt-out rise came from light users	More reminders buy lessons with permission you cannot get back; timing, not volume, is the lever (B4)
L4	Require a card to start the trial (30,000 paywall users per variant)	Stop	Paid per paywall user 11.10 to 10.12 percent, minus 0.98 points (minus 1.47 to minus 0.49)	Trial starts down 8 points	Monthly vs annual intent: both negative	Trial to paid rose from 37 to 46 percent, but fewer people started; judge trial changes on paid per eligible user, never on trial to paid alone (B5's guardrail)
L5	Streak counter on the home screen (10,500 per variant)	Ship, early stop at day 7	Next-day return 42 to 45 percent, +3.0 points (+1.66 to +4.34)	Lesson length flat	New vs existing users: larger for new	Flag: early stop, likely overstated; streaks matter enough to test the repair offer (B3)
L6	Shorter placement test, 10 questions instead of 20 (9,000 per variant)	Ship	Completion 62 to 65 percent, +3.0 points (+1.59 to +4.41)	Level changes in week 1 flat	Desktop vs mobile: mobile larger	Users tolerate the test when it is short; removing it entirely (B6) is a small, cheap step beyond this

Every backlog item above traces to at least one log row. L3's breach is why B4 changes timing rather than adding volume. L4 is why B5 must use paid per eligible user as a guardrail. L2 is the reason B2's 1.5-point MDE was a deliberate stretch: the team knows from its own interval that onboarding changes rarely move day-7 retention by more than a point, so B2 is either a bolder change than L2 or it is underpowered, and the row should say which.

L5 carries a flag for a reason. A test that clears an early sequential boundary tends to report a larger effect than the true one, which is covered with numbers in can you stop an A/B test early. Do not use a flagged effect to size the next test on that surface.

How do past tests become priors for the next ones?

Two numbers from the learning log replace guesswork in the backlog: your win rate and your typical effect size per surface.

The win rate sets the prior column. Ron Kohavi, Alex Deng and Lukas Vermeer argue in A/B Testing Intuition Busters that "in companies running online controlled experiments at scale, we can construct good prior estimates based on historical experiments," and they collect published success rates that run from 33 percent at Microsoft to 15 percent at Bing, about 10 percent at Booking.com, Google Ads and Netflix, and 8 percent for Airbnb search. Those rates come from different accounting schemes and very different products, so they set expectations rather than targets. Your own log is the number to use.

The win rate also tells you how much to trust a single win. Using the paper's formula (80 percent power, p below 0.05 two-sided, counting only significant improvements), a team whose log shows 9 wins from 41 tests, a 22 percent success rate, should expect about 10 percent of its significant wins to be false positives. At an 8 percent success rate that rises to 26.4 percent, the figure the paper gives for Airbnb search. A low win rate is a reason to replicate surprising wins before building on them.

Typical effect size sets the MDE column. If every shipped onboarding test in your log moved day-7 retention by 0.4 to 1.2 points, an onboarding idea sized to detect 3 points will almost always come back inconclusive. Either size it for the effects your surface actually produces and accept the longer run, or design a bolder change.

How do you run the quarterly experiment review?

Block 90 minutes each quarter with the PM, the analyst and an engineer who built at least one variant. Work through these steps in order:

1. Close the log. Every test that finished this quarter has a complete row, including the decision row and what the loser taught. Rows missing an interval are fixed before anything else.
2. Recount the win rate overall and per surface, and note the count behind each rate. A rate built on fewer than five tests borrows the team-wide rate instead.
3. Compare predicted with observed effects. For each finished test, put the MDE next to the measured effect. A surface where measured effects keep landing below the MDE needs smaller MDEs or bolder ideas.
4. Re-score the backlog with the new priors and effect sizes. Ideas whose score dropped below the cheapest shipped-without-test option move to "ship behind a flag" or are deleted.
5. Retire repeat ideas. Any backlog item that restates a hypothesis already in the log, with no new reason, is closed with a link to the row that answered it.
6. Pick one learning to share. One paragraph to the wider product team on the result that most changed what you believe, with the log row linked.

Step 5 is where the compounding shows up. Kohavi's team ran a weekly email of one interesting experiment for the same purpose; a one-paragraph quarterly note is the small-team version.

Is a learning log worth the overhead for a small team?

The fair objection: a team running two tests a month has six rows a quarter, too few for stable win rates, and the review takes time it could spend shipping. Both points are true early on. A per-surface win rate from four tests is noise, and the review checklist above already tells you to borrow the team-wide rate in that case.

The log pays off before the statistics do. The most common waste in a young testing programme is not a bad estimate; it is the same idea coming back six months later from someone who was not in the room, and an argument nobody can settle because nobody wrote down the interval. A row per test, written in ten minutes, prevents both. The priors arrive later, once there are enough rows to count, and by then the habit is in place. If your traffic is too thin for even one fixed test a month, multi-armed bandit vs A/B testing covers when an adaptive method is the better use of it.

Where can you build an experimentation habit like this?

Builders Camp runs A/B Testing for Product Managers as a 1 week bootcamp, live and self-paced, whose published topics include experimentation at scale: building an experiment backlog, documentation and learning compounding, alongside experiment design and when to ship, iterate or kill.

See the A/B Testing for Product Managers bootcamp

A useful first move this week: take your last five finished tests and write a learning log row for each, intervals included. If you cannot find the interval for three of them, the review ritual has already found its first job.