What an A/B test is
An A/B test compares two versions of something to see which performs better on a measure you chose in advance. Version A is usually the current one and version B has a single change, such as a different headline, button label or email subject line. Visitors or recipients are divided into two groups, each group sees one version, and you compare the outcomes.
The appeal is that it replaces opinion with evidence. The catch is that the evidence is statistical, which means it can be noisy, and a careless test can point confidently in the wrong direction.
Why the split must be random
If you showed version A on weekdays and version B on weekends, you would not know whether any difference came from the change or from the different kinds of people who visit on those days. Random assignment spreads those other differences, such as device, location or mood, roughly evenly across both groups. The two versions should also run at the same time, so that holidays, news events or a sudden traffic source affect both equally.
Most testing tools handle randomization and remember which version each visitor saw, often using a stored identifier, so that the same person does not see both. Because this relies on recognizing a browser, it is touched by the same limits as tracking cookies.
Changing one thing at a time
If version B differs from A in the headline, the image and the button, and B wins, you will not know which change mattered. Keeping the change to one element makes the result interpretable. Testing many changes together is possible but needs more traffic and more careful design.
Decide the question before you start
Before launching, write down three things:
- The change you are testing.
- The primary measure, such as the share of visitors who sign up, or the share of email recipients who open a message. A measure like a conversion rate must be defined clearly, including what counts as a conversion.
- When you will stop, based on a planned amount of data or time rather than on how the chart looks.
Choosing the measure after seeing the results invites you to find a “winner” in whatever happened to move. If you check many measures, some will differ by chance alone.
Sample size: why small tests mislead
Imagine flipping a coin ten times. Getting seven heads is not unusual even though the coin is fair. Small samples bounce around, and a small A/B test behaves the same way. If version A has 3 sign-ups from 40 visitors and version B has 5 from 40, that looks like a big improvement, but with so few people it could easily be luck.
A larger sample smooths out chance. How large depends on how common the outcome is and how small a difference you want to detect. As a general rule, the lower the baseline rate and the smaller the improvement you hope to notice, the more visitors you need, and the numbers can run into thousands per version. Many testing tools and online calculators estimate this for you. If your site simply does not get enough traffic, a formal test may not be practical, and it may be better to make a sensible change and watch the trend.
Why time matters
Behavior varies through the week. People shop and read differently on a Monday morning than on a Saturday night. A test that runs for only one day may capture an unusual slice of visitors. Running a test for at least one full weekly cycle, and often longer, gives a fairer picture. It also allows for the “novelty effect,” where something new gets extra attention at first and then settles.
Don’t peek and stop early
A common mistake is checking the results every day and ending the test the moment one version pulls ahead. Early on, differences swing widely, and repeatedly checking until something looks significant makes false winners much more likely. Decide the duration or sample size in advance and stick with it.
Reading the result
Tools often report a “statistical significance” or “confidence” figure. In plain terms, it is a rough indication of how likely it is that a difference this large would show up by chance if the two versions were really identical. It is not a guarantee, and it does not tell you the improvement will persist. Consider:
- Size of the difference. A tiny gain may not justify the effort of changing.
- Whether it holds across groups. A change that helps on desktop but hurts on mobile deserves a closer look.
- Side effects. A version that gets more clicks but leads to more returns or complaints may not be better overall.
- Whether the result is repeatable. If a result matters, running the test again can add confidence.
A test with no clear winner is still a result. It suggests the change did not matter much, which saves you from spending more effort on it.
Good candidates for testing
Pages with steady traffic and a clear goal are the best fit: a landing page, a checkout step that contributes to cart abandonment, an email subject line, a call-to-action wording, or a short section of product descriptions. Changes that affect something broader, such as page speed, are often better evaluated by comparing before and after periods, with caution about other things that changed in between.
Ethics and caution
Test changes that are honest. Showing misleading claims or hiding information to push people toward an action may increase a number while damaging trust. If tests involve personal data, make sure your privacy policy covers it.
Finally, keep modest expectations. Most tests produce small differences or none at all. The value of A/B testing is a habit of checking assumptions fairly, not a promise of dramatic gains.