Best A/B Testing Resources for Product Managers

Why A/B testing matters for product managers

A/B testing is the one growth skill where a wrong call is easy to defend and still be wrong. Builders Camp's A/B Testing for Product Managers bootcamp builds its entire practical challenge around exactly that trap: a Booking.com test where the variant wins on conversion rate at 95 percent confidence while a guardrail metric, average booking value, drops 8.6 percent and add-on revenue collapses 29 percent. The bootcamp's engineering lead wants to ship on the headline number alone. The challenge is teaching PMs to do the revenue math first.

That is the real skill this bootcamp's two live sessions and eight microlessons build: writing a testable hypothesis in a fixed format, choosing a primary metric alongside guardrails, and reading a results table without letting one green cell override three yellow or red ones. The certification quiz includes a direct question on why stopping a test early produces misleading results, which is the same mechanism Evan Miller's article below walks through in detail. None of this is optional theory. It is the difference between a PM who can defend a ship or kill decision with numbers and one who repeats whatever the dashboard says first.

Which articles explain the mechanics of a trustworthy test?

• How Not to Run an A/B Test, Evan Miller. The article that named the peeking problem: checking significance repeatedly during a test raises your real false positive rate to 26.1 percent, not the 5 percent most dashboards imply.
• How Many Subjects Are Needed for an A/B Test?, Evan Miller. A free, interactive sample size calculator, built by the same author, for planning a test's duration before you start rather than guessing at it.
• What is A/B Testing?, Optimizely. The plain-language definition and mechanism, useful as a first read before the sharper failure-mode pieces above.

Where can PMs hear real experimentation war stories?

• How We Put Facebook on the Path to 1 Billion Users, Chamath Palihapitiya. An annotated transcript of Palihapitiya's talk on the one behavioral signal (7 friends in 10 days) that Facebook's growth team found and rebuilt the product around.
• Ronny Kohavi on A/B Testing Done Right, Lenny's Podcast. Kohavi ran experimentation platforms at Microsoft, Amazon, and Airbnb; this conversation covers a Bing test that moved revenue 12 percent and when not to A/B test something at all.

How is AI changing A/B testing?

• Using AI for AI Testing, HubSpot. A practical rundown of where AI actually helps in an experimentation workflow, cleaning data and drafting variant copy, and where it does not replace the ship or kill call.

How Builders Camp teaches A/B testing

The A/B Testing for Product Managers bootcamp runs as a single, focused week: 2 live sessions, 3 hours of teaching, and 8 self-paced microlessons covering hypothesis writing, experiment design, statistical intuition, and interpreting mixed results. It sits inside both the Growth Specialist Track and the Data & Analytics Specialist Track, because reading a test correctly is a skill both disciplines depend on. The practical challenge puts members inside the Booking.com scenario above: reconstruct the hypothesis, classify each metric as green, yellow, red, or grey, calculate the actual revenue impact per 10,000 visitors, then write a direct reply to an engineering lead pushing to ship on a partial read. See the A/B Testing for Product Managers bootcamp for the full syllabus and current cohort dates.