All 29 modules are open from the start — nothing here is
locked, and nothing costs anything. Sign in so your progress, titles and
credentials stay with you, on every device you use.
Free forever, with your Google account. No password, no payment.
An A/B test splits traffic between two versions and measures which one performs better on a metric you chose in advance. Simple in concept, easy to get wrong in practice, this guide covers how to design a test you can trust and read its results honestly.
Ready?
1
What A/B Testing Is (and Isn't For)
An A/B test is a controlled comparison. You show version A to one randomly chosen group and version B to another, at the same time, and measure which performs better on one metric you decided in advance. The randomisation is what makes it work: it is the only thing that makes the two groups comparable.
That is the boundary. A/B testing answers which of these options performs better. It cannot tell you why, and it cannot generate options — it only ever compares the ones you already thought of.
So it fits some questions and not others. It needs enough traffic to detect a difference, an isolated change so you know what caused the result, a single complete metric chosen beforehand, and a decision that is reversible. Big strategic bets, low-traffic surfaces and “why do people churn?” each fail at least one of those.
Reaching for a test on a question it cannot answer is how teams spend six weeks learning nothing — and the failure is invisible, because the test still produces a number at the end.
Good fit
Isolated, well-defined change
Enough traffic, a complete single metric, a button, a flow, a headline.
Poor fit
Big strategic bets, tiny traffic, "why" questions
A/B testing tells you which option wins, not why, and needs real volume to work.
A test compares the options you already thought of. It can tell you which of two checkouts is faster, and never whether the shop should sell different food.
Everyday example
Two queues at a supermarket. You want to know which checkout is faster, so you send half the shoppers to each and time them. What you cannot learn this way is whether the shop should sell different food — a test compares two versions of the same thing, it does not tell you what the thing should be.
Google and the forty-one blues
Google famously tested dozens of shades of blue for its link colour rather than letting a designer pick one. It is the standard illustration of what A/B testing is for: settling a narrow, high-traffic, reversible question that opinion cannot settle. It is also the standard illustration of the limit — no shade of blue was ever going to tell the team what to build next.
Quick check
A team wants to A/B test whether to pivot the entire company to a new business model. Is A/B testing the right tool?
2
Designing a Test You Can Trust
Most tests that produce a wrong answer were designed wrongly, not analysed wrongly — and the design decisions all happen before anything ships.
Change one thing. If B differs from A in four ways and wins, you have learned that some combination of four things is better, and you cannot tell which — or whether one of them is doing harm that the others are hiding.
Pick the metric before you start, and pick one. Choosing afterwards from everything you collected guarantees you will find something that moved, because with enough metrics something always has.
Calculate the sample size in advance, and with it the date you will stop. This is the step people skip, and it is the one that makes every later decision honest, because it removes the temptation to stop when the line looks good.
A drug trial gives half the patients a placebo, assigned at random, and neither patient nor doctor knows which. Every one of these rules exists for the same reason: to make sure the only difference between your two groups is the thing you are testing.
1
Pick the randomisation unit
Usually the user, not the session — the same person seeing both versions on different visits is not a comparison.
2
Change one thing
If B differs from A in four ways and wins, you have learned that some combination of four things is better, and not which — or whether one is doing harm the others are hiding.
3
Pick one metric, in advance
Choosing afterwards from everything you collected guarantees you find something that moved, because with enough metrics something always has.
4
Calculate the sample size
Before anything ships. This is the step people skip, and it is what makes every later decision honest.
5
Set the stop date
Fixed at the same time as the sample size, so there is no temptation to stop on the day the line looks good.
The first three protect the comparison; the last two protect you from yourself. Every one of them has to be settled before anything ships — a sample size chosen on day four is a sample size chosen by looking at the result, whatever the arithmetic says afterwards.
Everyday example
A drug trial gives half the patients the real pill and half a placebo, decided at random, and neither the patient nor the doctor knows which. Randomisation is what makes the two groups comparable; without it you are comparing the people who chose the pill with the people who did not, and choosing is itself a difference.
Splitting on the wrong unit
A team ran a test on a collaboration feature and randomised by session, so the same person saw the old design on Monday and the new one on Tuesday, and coworkers in one shared document saw different things. The result was noise. The unit of randomisation has to be the unit the effect travels through — usually the user, sometimes the whole account.
Quick check
A team changes both the headline and the button color in their "B" variant, versus the original "A." What's the design flaw?
3
Statistical Significance and Common Pitfalls
Statistical significance is the formal answer to “could this difference have happened by chance?” A result is significant when the difference is large enough, and the sample big enough, that pure luck is an unlikely explanation. The conventional threshold is a 5% chance of a false positive, which people write as p < 0.05.
That threshold is worth understanding rather than reciting. It means that if you ran twenty tests where nothing was really happening, roughly one would still look significant. Run enough experiments and you will produce impressive false results on schedule.
Two related traps: too small a sample cannot detect a real effect, so “no difference” may mean “not enough data”. And a novelty effect means existing users react to change itself, which fades — so a two-day win among long-standing users may be measuring nothing but surprise.
The defence for all of this is the same: decide the sample size and the end date before you start, and then do not look.
Pitfall
Peeking
Checking results early and often, stopping the moment it looks significant, inflates false positives.
Pitfall
Novelty Effect
Users react to anything new, temporarily, early lift can fade once the change stops feeling new.
Pitfall
Sample Ratio Mismatch
If the actual split isn't the intended 50/50, something in the setup is likely broken.
Both labelled dots are real measurements, and both are wrong. The line did not become trustworthy because you waited — it was never trustworthy early, and the only difference between the honest reader and the other kind is which day they had already agreed to look.
Everyday example
Flip a fair coin ten times and you will sometimes get eight heads. Nobody concludes the coin is bent. Peeking at an experiment on day two and stopping because the line is up is exactly that conclusion — the run is short enough that chance can produce the whole effect.
Why the sample size goes in the plan
A team declared a 12% lift on day three, shipped it, and watched the metric return to baseline over the following month. Nothing had gone wrong with the instrumentation: they had stopped at a peak that the remaining sample would have flattened. Fixing the sample size and the stop date before the test starts is what removes the temptation, because by then the decision is already made.
Quick check
A team checks their test results every morning and stops the test the moment it crosses statistical significance on day 3 of a planned 4-week test. What's the risk?
4
Reading and Acting on Results
A result arrives: version B is up 3%, and it is statistically significant. That is the beginning of the work rather than the end of it.
Read the number honestly first. A test does not produce a fact, it produces a range.
Then ask whether the effect has any reason to persist. Existing users react to change itself, and that reaction fades — so a win measured over a few days among long-standing users may be measuring surprise rather than improvement. Name the mechanism that would make the effect durable. If you cannot, treat the result as provisional and plan to re-measure.
Then check the guardrails. Conversion up and refunds up is not a win. Almost any metric can be improved by damaging something it does not measure, so look at the numbers you agreed to protect before celebrating the one you were trying to move.
Finally, a flat result is a real result. It tells you this lever does not move this metric at this size, which stops you building three more versions of the same idea. The only genuinely wasted test is one whose outcome would not have changed what you did next — and that is worth establishing before you run it rather than after.
Every line on the right is what the line beside it looks like when nobody is being careless — they are the readings a pleased team arrives at honestly. The fourth pair is the one that costs most: a flat result filed as a failure is the finding that would have stopped three more versions of the same idea.
Everyday example
A weather forecast of "70% chance of rain" is not a promise. Neither is a test result: it is a range, and the honest sentence is "somewhere between a 1% and a 6% lift, probably around 3%". A single number quoted to two decimal places is a false precision that will be argued with later.
A lift that fades
A redesigned homepage shows a 15% signup lift in its first three days, then shrinks to
2% by week three. That's the novelty effect: users reacted to the new look, then
settled back to normal behavior. Run tests long enough to see past the spike, and
segment results, an aggregate win can hide a loss for a specific user group.
Statistical significance isn't the same as practical significance. A tiny but "real"
lift might not be worth the complexity of shipping it.
Acting on a result that was already gone
A homepage redesign showed a 15% lift and shipped. Three months later sign-ups were where they started. The novelty had worn off, and the test had run for exactly as long as novelty lasts. Reading a result includes asking whether the effect has a reason to persist.
Quick check
A test result is statistically significant with a 0.3% lift in conversion, but implementing the winning variant permanently would add real long-term maintenance complexity. What should the team weigh alongside statistical significance?
Drill what you learned
Scenario 1
easy
A team wants to test a new checkout flow but only gets 50 visitors a week to that page.
What's the concern?
Scenario 2
easy
A team runs a test with three different variants (A, B, and C) simultaneously, splitting traffic three ways.
Is this a valid design?
Scenario 3
easy
A test's control group gets the old checkout flow, and the variant group gets a new checkout flow that also happens to load on a faster server.
What's the design problem?
Scenario 4
easy
A team observes their test's traffic split is landing at 62/38 instead of the intended 50/50.
What should they do?
Scenario 5
easy
A test reaches p < 0.05 after just 200 total visitors, far below the pre-calculated sample size of 5,000.
Should the team trust this early result?
Scenario 6
medium
A redesigned homepage shows a 15% lift in signups during its first three days, then the lift shrinks to 2% by week three.
What's the likely explanation?
Scenario 7
medium
A PM wants to know "why" users prefer variant B over variant A, not just which one performs better.
What should supplement the A/B test?
Scenario 8
medium
A team pre-registers success criteria (95% confidence, minimum 3% lift) before running a test, then hits 95% confidence with only a 1% lift.
What should they do?
Scenario 9
medium
A test is stopped at the pre-planned sample size, but the result shows the variant performing worse than control, with high confidence.
Is this test a failure?
Scenario 10
medium
A team segments their test results and finds the variant won overall, but actually performed worse specifically for mobile users.
What should the team do with this finding?
Scenario 11
hard
A stakeholder wants to run a test on a page that gets huge traffic but insists on ending it after just 6 hours because "we already have enough data."
What's the concern, even with high traffic?
Scenario 12
hard
A team's variant wins with statistical significance, and someone proposes immediately testing five more small tweaks on top of it before locking in the winner.
What's the risk with this plan?
Scenario 13
hard
A test's variant is a completely redesigned page with a new layout, new copy, new images, and a new checkout flow, all shipped as a single test.
What's the core design flaw here?
Scenario 14
hard
A team wants to declare victory on a test that reached significance, but they never decided in advance how long to run it or how large a sample they needed.
What's missing from their process?
Scenario 15
hard
A test shows a statistically significant result, and a PM says "ship it" without checking whether the lift is large enough to justify the engineering cost of maintaining two code paths going forward.
What step did the PM skip?
Drilled it. Now apply it to a real situation.
Put it to work
From lesson 1
Bing's headline change, and why nobody believed it
Microsoft Bing
An engineer at Bing suggested moving the ad headline onto the same line
as the site name. It sat in the backlog for six months: low effort, but
nobody thought it mattered.
When it finally ran as an experiment, revenue rose sharply — the change
turned out to be worth in the region of $100 million a year. Ronny
Kohavi, who ran experimentation there, has used it ever since as the
example of why teams cannot rank ideas by intuition.
What does this case actually establish about A/B testing?
Select all that apply — there are 3 to find.
What actually happened
Bing built one of the largest experimentation platforms in the industry
on the back of this thinking, running thousands of concurrent tests.
Kohavi's published figures are blunt: roughly a third of well-designed
experiments produce a positive result, a third do nothing, and a third
actively hurt.
That last third is the real argument for testing. It is not that you
find wins faster; it is that you stop shipping losses you would never
have noticed.
From lesson 2
The test that measured the wrong thing
A meal-kit subscription
The team wants to know whether a slimmer signup flow gets more people
subscribing. They cut four steps to two and run it for a week.
Variant B wins on signups by 18%, comfortably significant. The team is
ready to ship. Before they do, you look at how the test was built.
How the test was set up
Decision
What they did
Assignment
By session, not by user
Traffic
B shown to weekend visitors, A to weekday
Duration
7 days
Primary metric
Signups started
Guardrail
None
Two of these five make the result unusable on their own.
Which choices make this result untrustworthy?
Select all that apply — there are 3 to find.
What actually happened
Rebuilt properly — random assignment by user, both variants running
across the whole week, primary metric on completed subscriptions and a
guardrail on first-month cancellation — the effect was +4%, not +18%.
Real, worth shipping, and a quarter the size of the original claim.
A test you cannot trust is worse than no test, because it comes with a
number attached and nobody argues with a number.
From lesson 3
Stopping the moment it looks good
A B2B analytics tool
The team is testing a new pricing page. They agreed up front on a
two-week run. On day three the dashboard shows B ahead with p = 0.04,
and someone proposes calling it.
The same experiment, checked daily
Day
Conversion A
Conversion B
p-value
3
3.1%
4.0%
0.04
6
3.3%
3.7%
0.21
9
3.4%
3.5%
0.58
14
3.4%
3.4%
0.91
Work through what the table is telling you.
Step 1 of 3
What happened between day 3 and day 14?
Had they stopped on day 3, they would have shipped a change worth nothing and booked a 29% lift that never existed.
Step 2 of 3
Why does checking the p-value every day make this worse?
The fix is not to avoid looking. It is to fix the stopping rule in advance and hold to it.
Step 3 of 3
What should have been agreed before the test started?
Significance answers "could this be noise". It never answers "is this big enough to matter" — that is a decision you make before the test, not after it.
What actually happened
They ran the full two weeks and shipped nothing. The pricing page test
that followed — a genuine restructure rather than a copy change — moved
conversion 0.6 points and held up across the whole run.
The discipline that made the second result believable was the one that
made the first result boring: the stopping point was set before anyone
knew which way the numbers would go.
From lesson 4
A win on the primary metric, a loss everywhere else
A news app
The team tested a more aggressive push-notification schedule: three a
day instead of one. The primary metric was daily active users.
Two-week result, variant B vs. control
Metric
Change
Significant
Daily active users
+9.4%
Yes
Sessions per user
+11%
Yes
Notification opt-outs
+62%
Yes
Day-30 retention
-3.1%
Yes
App store rating
4.4 → 4.1
n/a
Order these responses from best to worst.
Drag the rows, or use the arrows, then check.
Do not ship. The guardrails say this buys short-term activity with long-term users.First. Opt-outs and day-30 retention are both significant and both point the same way: the lift is being paid for out of the future. The primary metric won and the product lost.
Test a middle option — two notifications a day, better targeted — with the same guardrails.Second. The result is informative even though it is a loss: it says frequency has real upside and a real ceiling. Finding the ceiling is a reasonable next test.
Ship it, but commit to watching retention monthly and rolling back if it worsens.Third. Better than shipping blind, but retention damage is slow and hard to attribute after the fact, and you already have the evidence in hand.
Ship it. DAU was the primary metric and it won significantly.Last. This is what guardrails exist to prevent. A pre-declared primary metric settles which number decides a close call — it does not license ignoring every other number on the table.
What actually happened
They did not ship. The follow-up test — two notifications, and only for
stories in topics the reader had actually opened before — took +5% on
daily actives with opt-outs flat and retention unchanged. Smaller
headline, and the only one of the two that was real.
Declare the primary metric and the guardrails before the test. The
primary metric decides a close call; the guardrails decide whether you
are allowed to ship at all.