Multi-armed bandit vs A/B testing: which should a PM use?

The common pitch for bandits is that they reach decisions faster. The vendors who build them are more careful than that. Optimizely's documentation on multi-armed bandit optimizations states the trade-off directly:

"Because fixed traffic allocations are optimal for reaching statistical significance, MAB-driven experiments generally take longer to find winners and losers than A/B tests," Optimizely's support documentation explains.

A bandit is not a faster A/B test. It answers a different question. An A/B test asks which variant is better and by how much. A bandit asks how to collect the most conversions while you are still finding out. Amplitude's multi-armed bandit documentation shows how differently the two treat confidence: with a 95 percent confidence level, once the bandit has sent at least 95 percent of traffic to one variant, Amplitude assumes confidence and sends it 100 percent from then on. That is a rule for committing traffic, not a measurement of effect size.

What is the difference between a multi-armed bandit and an A/B test?

An A/B test splits traffic in fixed proportions, usually 50/50, and keeps that split until it reaches a sample size planned in advance. At the end you get an estimate of the difference between variants, a confidence interval around it, and a clean comparison against a control group. The A/B testing glossary entry covers the basics.

A multi-armed bandit starts with an even split and then moves traffic toward whichever variant looks stronger, over and over, as results arrive. It balances two pulls: exploration (keep showing every variant enough to learn about it) and exploitation (send people to the current leader). Amplitude's implementation, for example, uses Thompson sampling, can reallocate hourly, daily or weekly, and waits for at least 100 exposures per variant before its first reallocation.

The cost a bandit minimises has a name: regret, the conversions lost by showing users a variant that is worse than the best one. A 50/50 test runs up regret on purpose for the whole test to buy a clean answer. A bandit cuts regret as soon as the evidence tilts, and pays for it in measurement quality.

Is a bandit faster than an A/B test?

Whether a bandit is faster depends on what "faster" means, and the answer changes the decision. A bandit is faster at moving users to a probable winner, which matters when every day of the window counts. It is slower at establishing that the winner really is better, because the losing variants get fewer and fewer users, so their estimates stay noisy. Optimizely states plainly that bandits do not generate statistical significance and do not use a control or baseline experience.

For a PM, that means a bandit cannot answer "how much better is the new onboarding than the old one?" If that number goes into a business case, a roadmap argument or a pricing decision, you need an A/B test.

Which questions decide between a bandit and an A/B test?

Five questions settle most cases. Answer them before you pick a tool:

Question	Points to a bandit	Points to an A/B test
How long does the change live?	Days or weeks (a campaign, a headline, a seasonal offer)	Permanent (a flow, a default, a price)
How fast does the success metric register?	Minutes to hours (click, open, add to cart)	Days to weeks (retention, renewal, refunds)
What does a bad variant cost?	Little: a weaker subject line	A lot: lost revenue, trust, or a support spike
Do you need to know why and by how much?	No, you want the most conversions in the window	Yes, the effect size feeds a decision or a story
Are there metrics that must not get worse?	No real guardrails	Yes, and you want to watch them with a fair comparison

The fourth and fifth rows are the ones product teams underrate. Amplitude's documentation is blunt about guardrails: a bandit optimises one primary metric, secondary metrics are reporting only, and "If you face a tradeoff between metrics you want to optimize, run an A/B test instead." Most product changes that matter carry exactly that kind of trade-off. The guide to running an A/B test shows how guardrails fit into a fixed test.

What does a small-sample case look like?

Take two original scenarios from the same company, a language learning app.

The growth team wants to pick one of four push notification texts for a two-week spring promotion aimed at lapsed users. The metric (a tap that opens the app) arrives within minutes, the content expires with the promotion, and a weaker text costs a few taps. A fixed four-way split would keep sending three quarters of lapsed users to losing texts for the whole promotion. That is textbook bandit territory, and the Optimizely glossary on multi-armed bandit testing lists headlines and short-term campaigns as the typical fit.

The product team wants to shorten the default lesson from ten minutes to five. The outcome that matters, whether learners are still practising a month later, takes weeks to register, the change is permanent, and a wrong call damages the habit the whole product depends on. A bandit would reallocate on early engagement, which shorter lessons will almost certainly win, and never see the retention effect. That is an A/B test with a long enough run, or a holdout kept for months.

Same app, same team, opposite answers, because the time window, the metric delay and the cost of error differ.

How does personalization break standard A/B test conditions?

A standard A/B test assumes each group gets one fixed treatment, and that the treatment means the same thing on day one and day twenty. Personalization breaks both assumptions on purpose. If every user sees a recommendation list tailored to their history, "variant B" is not one thing, it is thousands of different screens, and the system producing them changes as it learns.

Contextual bandits go one step further: they choose a variant per user based on that user's context and keep adapting. Optimizely's glossary describes its contextual bandits as starting with full exploration and keeping some exploration running (at most 95 percent exploitation) so they keep learning.

The workable fix is to change what you test. Test the policy, not the outputs: randomly assign users either to the personalization system or to a fixed default, and compare the groups on the metric you care about, over a long enough window to see retention. That returns you to a clean A/B comparison of whether personalization as a whole earns its complexity, while the bandit handles the per-user choices inside the treated group.

What does a bandit cost you that is easy to miss?

Three costs show up after the fact. First, delayed metrics: a bandit can only optimise what it observes quickly, so it drifts toward whatever wins early (clicks) even when the goal is later (purchases, retention). Second, time effects: if the leader on a Monday is not the leader on a Saturday, early reallocation can lock in a variant for the wrong reason. Third, lost learning: with no stable control and thin data on losing variants, there is little to write in the experiment log beyond "this one got the traffic".

None of these rule bandits out. They mean a bandit belongs where a quick, reversible choice is the whole job.

Isn't every 50/50 test wasting traffic on losers?

Wasted traffic is the strongest argument for bandits, and on short-lived content it is correct: running a losing headline to half your audience for two weeks is a real cost with no lasting benefit. On a permanent change, the argument flips. The traffic "wasted" on the control is what buys an unbiased estimate of the lift, and a permanent change without that estimate is a decision made on a guess that will be paid for every day after launch. Whether that lift is worth shipping is a further question, covered in statistical vs practical significance.

Where can you learn to pick the right experiment design?

Builders Camp runs A/B Testing for Product Managers as a 1 week bootcamp with 2 live sessions and 8 self-paced microlessons, directed by Andre Albuquerque. Its public syllabus covers experiment design (variants, guardrails and duration), metric selection, statistical intuition for PMs, interpreting results, and experimentation at scale.

See the A/B Testing for Product Managers bootcamp

A practical first step: take your last five experiments and sort them with the five questions above. If most land in the A/B column, a bandit tool is not your bottleneck; if several were short-lived content tests that ran as 50/50 splits, that is where a bandit earns its place.