---
title: "Can You Stop an A/B Test Early? The Rules"
description: "Can you stop an A/B test early? Daily peeking turned 5 percent false winners into 25 percent in our simulation. Two legitimate early-stop designs, and a policy."
canonical_url: "https://builderscamp.com/guides/other/can-you-stop-an-ab-test-early"
date_published: "2026-10-05"
date_modified: "2026-10-05"
author: "Joy Chatterjee"
publisher: "Builders Camp"
guide_class: "other"
---

# Can you stop an A/B test early? Only with a stopping rule you wrote before launch

**TL;DR:** Not with a standard fixed-sample test. Checking a no-difference test daily for 21 days and stopping at the first p below 0.05 produced a false winner in 25.2 percent of 20,000 simulated runs, five times the 5 percent you think you are accepting. Two early stops are legitimate: a sequential design whose interim thresholds are set before launch, and a guardrail rule that stops for harm, never for a win.

Every A/B testing tool shows a live p-value, and that number invites the wrong question. "Is it significant yet?" is a question a fixed-sample test can only answer once. Evan Miller's advice in [How Not To Run an A/B Test](https://www.evanmiller.org/how-not-to-run-an-ab-test.html) is blunt: "Decide on a sample size in advance and wait until the experiment is over before you start believing the 'chance of beating original' figures that the A/B testing software gives you."

The size of the problem has been measured more than once. Leonid Pekelis, introducing [Optimizely's Stats Engine](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/) in 2015, reported that when results were checked after every new visitor, "more than 57% of simulated A/A tests falsely declared a winner or loser at least once during their course." The rest of this page shows the same effect on a product-sized test, then the two designs that let you stop early without fooling yourself.

## Why does checking an A/B test every day create false winners?

A 5 percent significance threshold means that, if the change does nothing, a single reading at the planned end crosses the line 5 percent of the time. Read the test every day and stop at the first crossing, and each day is a fresh draw. Noise only has to cross the line once.

To put numbers on a realistic test, we simulated it for this guide. The setup: a checkout step that converts 10 percent of users, 2,000 users per variant per day, a planned 21-day run (42,000 users per variant), a two-sided z-test on the difference in conversion rates at 0.05, and 20,000 simulated tests in which both variants were identical. Any "winner" is therefore false.

| How the test was read | False winner or loser rate |
|---|---|
| Once, on day 21 as planned | 5.0 percent |
| Weekly (days 7, 14, 21), stop at first p below 0.05 | 10.8 percent |
| Daily (21 looks), stop at first p below 0.05 | 25.2 percent |
| Weekly, with sequential boundaries set before launch | 5.0 percent |

The daily figure lines up with published results that used different setups. Evan Miller's worked example (50 percent baseline, stop at the first 5 percent reading or after 150 observations) gets 26.1 percent. Pekelis reported 26 percent when results were checked every 500 visitors. The exact number depends on how many looks you take, and it only rises with more looks, which is the point: the error rate on the dashboard is not the error rate of your decision process.

## What is the difference between watching a test and deciding on it?

Watching is checking that the experiment is healthy. Deciding is letting the comparison between variants end the test. A team can watch every hour without inflating anything, as long as nothing it sees about the primary metric changes when the test ends.

| What you see on an interim look | What you do | Does it decide the hypothesis? |
|---|---|---|
| The split is 52/48 when you planned 50/50 | Stop, fix the assignment bug, restart with fresh users | No, the test was broken |
| An event stopped firing in one variant | Pause, fix tracking, restart the clock | No |
| Error rate or page load time spiking in the variant | Apply the guardrail rule if one exists; otherwise roll back as an incident | Only through a pre-registered rule |
| Primary metric shows p of 0.03 on day 9 | Note it; keep running | No, unless a sequential plan names this look |
| Primary metric looks flat on day 9 | Keep running | No; the test is underpowered until its planned size |

The bottom two rows are where teams slip. A flat interim result is as unreliable as a significant one, and stopping a test early "because nothing is happening" wastes the sample you already paid for.

## What are the legitimate ways to stop an A/B test early?

Two designs let a team stop before the planned end and still trust the error rate it chose. Both have to be written down before launch.

1. **A pre-registered sequential test.** You fix in advance how many interim looks you will take and a stricter threshold for each. Crossing a boundary at an interim look is a valid result for or against the variant.
2. **A guardrail breach rule.** You fix in advance a one-sided harm threshold on one or two guardrail metrics. Crossing it stops the test for harm. It can never declare a winner.

A third route is to use a platform whose statistics are built for continuous monitoring. Pekelis wrote that "Sequential tests can be stopped any time with valid results," and the [Statsig documentation on sequential testing](https://docs.statsig.com/experiments/advanced-setup/sequential-testing) describes adjusting "p-values and confidence intervals for each preliminary analysis window" to compensate for peeking. If your tool offers this, turning it on before launch counts as the first design. Turning it on after you have seen the numbers does not.

## How does a pre-registered sequential test work?

A group sequential plan spends the 5 percent error budget across a fixed number of looks, keeping the early thresholds very strict so that only a large effect can stop the test early. For the same 21-day test, we calibrated an O'Brien-Fleming style boundary for three weekly looks by simulating 2 million random walks. These are the thresholds:

| Look | Users per variant so far | Stop for a win or loss if z is beyond | Equivalent two-sided p below |
|---|---|---|---|
| Day 7 | 14,000 | 3.47 | 0.0005 |
| Day 14 | 28,000 | 2.45 | 0.014 |
| Day 21 | 42,000 | 2.00 | 0.045 |

On the 20,000 no-difference tests, this plan produced false winners or losers 5.0 percent of the time, the same as reading once. On 20,000 tests with a real lift from 10.0 to 10.7 percent conversion (7 percent relative), it caught the win 90.9 percent of the time against 91.3 percent for the fixed 21-day test, and the average test ran 16.4 days instead of 21. Most of the saving came at the second look: 54.1 percent of all tests stopped at day 14 and 6.1 percent at day 7.

The final threshold (p below 0.045) is barely stricter than 0.05, which is why the plan costs so little power. The day 7 threshold is the opposite: only a result far larger than the one you sized for can clear it.

If your stack has no sequential tooling, Evan Miller's [Simple Sequential A/B Testing](https://www.evanmiller.org/sequential-ab-testing.html) gives a rule that needs only counts: fix a total number of conversions N in advance, declare the treatment the winner if its conversions exceed control's by 2 times the square root of N, and stop with no winner once the two together reach N. The rule is one-sided (it can only crown the treatment) and works best at low conversion rates. When the test is sized for a 10 percent lift and that lift is real, he reports expected savings as high as 17.4 percent against a fixed-sample test.

## Do early stops overstate the effect?

They do, and it matters for whatever you build on the number. In the same simulation, with a true lift of 7 percent, the tests that stopped at day 7 reported an average lift of 14.7 percent. Those stopped at day 14 reported 8.5 percent, and the fixed 21-day test averaged 7.0 percent. A test can only clear a very strict early boundary when noise happens to push in the same direction as the effect, so the early winners are the lucky draws.

Statsig's documentation makes the same trade-off explicit: "Early decisions often produce underpowered lift estimates with high uncertainty," and "If you need an accurate measurement of the effect size, wait for the full power that your pre-experiment power calculation estimates." The working rule: an early stop answers "ship or not". It does not give you the number for the revenue forecast or for sizing the next test. Write the early-stopped estimate into your learning log with a flag, and expect the shipped effect to be smaller.

## How does a guardrail breach rule work?

A guardrail breach rule exists to protect users and revenue from a variant that is clearly causing harm, not to find winners faster. It is one-sided (it only looks for the guardrail getting worse), it uses a much stricter threshold than 0.05, and it covers one or two guardrails, not every metric on the dashboard.

In the simulation, the rule "stop if the guardrail's z is at or below minus 3 on any daily look" was tested two ways. On identical variants, it raised a false alarm in 1.1 percent of tests across 21 daily looks. On a variant that genuinely cut a 10 percent guardrail rate to 9 percent (a 10 percent relative drop), it caught the harm in 98.2 percent of tests by day 21, and in half of them by day 7.

The asymmetry is deliberate. A false harm alarm costs you a rerun. A false win ships a change that does nothing, or does damage, and nobody tests it again. That is why a team can afford a sensitive daily rule for harm and cannot afford one for wins.

## Which early-stop design should your test use?

Pick the design from the reason you might want to stop, before launch:

1. If the risk is harm (revenue, cancellations, errors, latency), add a guardrail breach rule on the one or two metrics that would hurt most, with a threshold of z at or below minus 3 or stricter.
2. If the pressure is opportunity cost (a winner should ship as soon as the evidence allows), add a sequential plan with a fixed number of looks, at most one per full week of traffic, or turn on your platform's sequential statistics before launch.
3. If the result will feed a business case or set the effect size for future tests, keep the fixed horizon for the primary metric, because you need the unbiased estimate more than the saved days.
4. If none of these applies, run the fixed test to its planned end and use interim looks only for health checks.

Seasonality sets a floor on all of them: do not let any boundary fire before one full week, because weekday and weekend users behave differently and a three-day sample is a sample of the wrong population. Even with statistics built for continuous checking, Optimizely's [Stats Engine guide](https://certification.optimizely.com/concepts/experimentation/stats-engine-deep-dive/) advises: "Set a minimum runtime of one to two business cycles to capture weekly patterns."

## What should you say when a stakeholder wants to ship at day 14 of 21?

Start from what was written down. The [A/B test results challenge](https://builderscamp.com/guides/challenges/ab-testing-for-product-managers-ab-test-results) puts you in exactly this spot: day 14 of 21, primary metric at p of 0.041, an engineering lead asking to ship.

If no interim look was pre-registered, p of 0.041 at day 14 is one reading out of many the team could have taken, and the dashboard's 5 percent error rate no longer applies. The test runs to day 21. If the team had adopted the weekly sequential plan above, the same reading would be checked against the day 14 boundary of p below 0.014, and it does not clear it. Either way the answer is the same, and it comes from the plan rather than from anyone's confidence.

A reply that works with a peer: "The 0.041 assumes we only look once, at the end. We have looked several times, so the real false-win risk is higher than it reads. Our plan says day 21, and two guardrails are still moving. If anything breaches the harm rule before then, we stop immediately." Then check the guardrails, because a primary win with a broken guardrail is not a ship decision at any sample size, as [statistical vs practical significance](https://builderscamp.com/guides/other/statistical-vs-practical-significance) works through.

## Isn't a sequential test just a slower way to ship?

The objection has real weight. A sequential plan adds work before launch, needs someone to calibrate or configure the boundaries, gives wider intervals at early looks, and overstates the effect when it stops early. Teams with plenty of traffic and short tests often gain little from it, and a fixed 14-day test read once is simpler to explain.

The case for it is the alternative that actually happens without it. Teams under deadline pressure peek anyway. A sequential plan turns that pressure into a rule with a known error rate: in the simulation, 60 percent of tests with a real 7 percent lift were stopped as wins by day 14, at the same 5 percent false positive rate and with almost no loss of power. If your tests regularly face ship-now pressure, the plan costs less than the arguments it prevents. For tests where you want to shift traffic toward the winner continuously rather than stop at fixed looks, [multi-armed bandit vs A/B testing](https://builderscamp.com/guides/comparison/multi-armed-bandit-vs-ab-testing) covers the adaptive alternative and what it gives up.

## What does a one-page early-stop policy look like?

Paste this into your experimentation doc and fill the brackets per test. It is short on purpose: a policy nobody reads before launch protects nothing.

```
EARLY-STOP POLICY (attach to every test plan, agreed before launch)

Test: [name]                     Owner: [PM]      Analyst: [name]
Planned sample: [users per variant]   Planned end: [date, whole weeks]

1. Health checks (any time, anyone)
   - Sample ratio within [49/51] of planned split
   - Key events firing in both variants
   - Error rate and load time within budget
   Breach -> pause, fix, restart with fresh users. Not a result.

2. Guardrail breach rule (daily, harm only)
   - Guardrails: [metric 1], [metric 2]
   - Stop if variant worse with z <= [-3.0] on any daily look
   Breach -> stop, roll back, log as "stopped for harm".

3. Interim decision looks (only if pre-registered)
   - Looks: [day 7, day 14, day 21]
   - Boundaries (two-sided p): [0.0005, 0.014, 0.045]
   - No look before one full week of traffic
   Crossed -> decide; log estimate as "early stop, likely overstated".

4. Everything else
   - Interim p-values on the primary metric are not decisions.
   - Flat interim results are not decisions.
   - Changing this policy after launch requires a restart.

Sign-off: PM [ ]  Analyst [ ]  Eng lead [ ]
```

The last line of section 4 is the one that does the work. Every exception a team grants itself mid-test is a look that was not in the plan, and the simulation shows how quickly those add up.

## Where can you practise these calls?

The pre-launch side of this policy (hypothesis, primary metric, guardrails and planned sample) is set out step by step in [how to run an A/B test](https://builderscamp.com/guides/other/how-to-run-an-ab-test). When the change has to launch to a whole market and no split is possible, [pre/post vs difference in differences vs A/B test](https://builderscamp.com/guides/other/pre-post-vs-difference-in-differences-vs-ab-test) covers what you can still claim.

Builders Camp runs A/B Testing for Product Managers as a 1 week bootcamp, live and self-paced, covering experiment design (variants, guardrails and duration, and how to avoid peeking and hidden bias) and when to ship, iterate or kill.

[See the A/B Testing for Product Managers bootcamp](https://builderscamp.com/bootcamps/ab-testing-for-product-managers?utm_source=guide&utm_medium=organic&utm_campaign=can-you-stop-an-ab-test-early)

Before your next launch, ask your analyst one question: "At which looks are we allowed to decide?" If the answer is "whenever it looks significant", you now know the real error rate is closer to 25 percent than 5.

## Frequently asked questions

### Can you stop an A/B test early if it is already significant?

Not with a standard fixed-sample test. In a simulation of 20,000 tests of a change that did nothing, checking every day for 21 days and stopping at the first p below 0.05 declared a false winner 25.2 percent of the time, against 5.0 percent when the same tests were read once at the planned end.

### What is peeking in A/B testing?

Peeking is checking a running test's significance repeatedly and being willing to stop the first time it crosses the threshold. Looking is harmless on its own. The damage comes from letting any interim look decide the test, because every look gives random noise another chance to cross the line.

### How often can I check a running A/B test?

As often as you like for health signals such as a broken split, missing events or error spikes. For the decision itself, only at the looks your plan named in advance, against the boundary set for each look, or on a platform whose sequential statistics are built to stay valid under continuous checking.

### What is a sequential A/B test?

A test designed from the start to be read more than once. It fixes the number of interim looks and a stricter threshold for each, so the overall false positive rate stays at the level you chose. With three weekly looks, the thresholds used on this page are p below 0.0005, 0.014 and 0.045.

### Is it ever right to stop a test because a guardrail is getting worse?

Yes, if the guardrail rule was written before launch: a one-sided check for harm only, with a strict threshold. In the simulation on this page, a rule of z at or below minus 3 on any daily look raised a false alarm in 1.1 percent of no-difference tests and caught a real 10 percent relative drop in 98.2 percent.

### Do early-stopped winners overstate the effect?

Yes, on average. In the simulation on this page, the true lift was 7 percent; tests that cleared the sequential boundary at day 7 reported an average lift of 14.7 percent. Use the early stop to make the ship decision, and use a longer-run estimate when you size the business case or the next test.

### What should I tell a stakeholder who wants to ship at day 14 of 21?

Tell them what the plan allows. If no interim look was pre-registered, the day 14 p-value does not mean what the dashboard implies, so the test runs to day 21. If a sequential plan exists, check the number against that look's boundary and follow it, whichever way it goes.

## Sources

- [Evan Miller: How Not To Run an A/B Test](https://www.evanmiller.org/how-not-to-run-an-ab-test.html)
- [Evan Miller: Simple Sequential A/B Testing (2015)](https://www.evanmiller.org/sequential-ab-testing.html)
- [Optimizely: Statistics for the internet age, the story behind Optimizely's new Stats Engine (Leonid Pekelis, 2015)](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/)
- [Statsig Docs: Sequential Testing](https://docs.statsig.com/experiments/advanced-setup/sequential-testing)
- [Optimizely: Stats Engine, how statistical decisions work](https://certification.optimizely.com/concepts/experimentation/stats-engine-deep-dive/)

## How this guide was made

Researched from Builders Camp's bootcamp, track and masterclass material and the sources listed on this page, drafted with AI, and fact-checked against every source cited.