--- title: "A/B Testing: Definition and PM Examples" description: "A/B testing compares two versions of a product to see which one performs better on a defined metric. See a real example and how the bootcamp teaches it." canonical_url: "https://builderscamp.com/guides/glossary/ab-testing" date_published: "2026-09-16" date_modified: "2026-09-16" author: "Andre Albuquerque" publisher: "Builders Camp" guide_class: "glossary" --- # What Is A/B Testing? **TL;DR:** A/B testing compares two versions of a product against a control group to see which one performs better on a defined metric, using random assignment and statistical significance to rule out chance. The hard part is not running the test, it is reading an ambiguous or mixed result correctly. ## What does A/B testing mean? A/B testing, also called split testing, is a methodology for comparing two versions of a product or page against each other to determine which one performs better against a defined goal. Per [Optimizely](https://www.optimizely.com/optimization-glossary/ab-testing), it works by showing two variants to users at random and using statistical analysis to determine which variation achieves better results, comparing conversions from the original version, the control, against the new version, the challenger. The method depends on random assignment and a large enough sample to trust the result, which is why [statistical significance](https://builderscamp.com/guides/glossary/statistical-significance) and a proper [control group](https://builderscamp.com/guides/glossary/control-group-experimentation) are not optional extras, they are what makes an A/B test different from just watching a metric change after a launch. For the step by step procedure, see [how to run an A/B test as a product manager](https://builderscamp.com/guides/other/how-to-run-an-ab-test); for when an adaptive method fits better, see [multi-armed bandit vs A/B testing](https://builderscamp.com/guides/comparison/multi-armed-bandit-vs-ab-testing). ## Why A/B testing matters for product managers A/B Testing for Product Managers is built around a specific gap the bootcamp identifies directly: most PMs can design an A/B test, but very few can read an ambiguous result under pressure and make the right call. The bootcamp teaches the full loop from hypothesis to decision, including how to avoid the two failure modes that produce false confidence, peeking at results too early and ignoring guardrail metrics that move the wrong way. The bootcamp is explicit that a mixed result is not a verdict either way. Learning to interpret partial or contradictory results honestly, rather than picking whichever number supports the outcome you wanted, is treated as the actual skill being tested. ## A/B testing example The bootcamp's practical challenge puts a PM at Booking.com midway through a 21 day test of a "Price Match Guarantee" badge. At day 14 of 21, the variant shows a 9.4 percent lift in booking conversion at 95 percent confidence, but average booking value is down and add-on purchases have dropped 29 percent, both moving the wrong direction. An engineering lead pushes to ship immediately based on the headline conversion number alone. The exercise requires calculating the actual net revenue impact per 10,000 visitors, weighing the conversion gain against the lost booking value and lost add-on revenue, before deciding whether to ship, kill, or extend the test to its planned full duration, which is exactly the kind of ambiguous, numbers first call real PMs face. ## How Builders Camp teaches A/B testing Builders Camp teaches A/B testing inside the [A/B Testing for Product Managers bootcamp](https://builderscamp.com/bootcamps/ab-testing-for-product-managers), directed by Andre Albuquerque, across two live sessions covering hypothesis design, [statistical significance](https://builderscamp.com/guides/glossary/statistical-significance), and interpreting mixed results under stakeholder pressure. Builders Camp runs live and self-paced bootcamps in product management and AI product building. [See the A/B Testing for Product Managers bootcamp](https://builderscamp.com/bootcamps/ab-testing-for-product-managers) for the next cohort dates. ## Frequently asked questions ### What is the primary purpose of A/B testing? To compare two or more versions of a product against each other to determine, with statistical evidence rather than opinion, which one performs better against a specific, predefined goal. ### What is a common mistake teams make with A/B tests? Stopping the test as soon as the primary metric looks significant, before the planned sample size or duration is reached, which produces misleading, unstable results. ### Does A/B testing replace product judgment? No. A/B testing tells you what happened under specific conditions, not why, and it still requires a clear hypothesis and a human decision about what the result means for the roadmap. ### What should happen if guardrail metrics move the wrong way during a test? The team should weigh the guardrail degradation against the primary metric gain in real revenue or user terms before shipping, rather than shipping on the primary metric's significance alone. ### Can two A/B tests run on the same users at once cause problems? Yes. Overlapping tests on the same audience segment can introduce interaction effects between the changes, making it hard to attribute a result to either test cleanly. ### How long should an A/B test run? Long enough to reach the sample size the test was planned for, covering enough full business cycles that day of week or seasonal effects do not distort the read. ## Sources - [Optimizely: A/B Testing](https://www.optimizely.com/optimization-glossary/ab-testing) ## How this guide was made Researched from Builders Camp's bootcamp, track and masterclass material and the sources listed on this page, drafted with AI, and fact-checked against every source cited.