A/B testing sounds simple: show half your visitors version A, half version B, and keep the winner. In practice, many tests produce "winners" that never improve revenue, or end inconclusively after weeks of traffic. The cause is rarely the idea being tested. It is usually the experiment itself: a vague hypothesis, too little traffic, stopping too early, or measuring the wrong thing.
Done properly, A/B testing is one of the most reliable ways to improve a website, an ad or an email, because it replaces opinions with evidence. It also protects you from confident redesigns that quietly reduce conversions.
This guide explains how A/B testing works, how to write a hypothesis worth testing, how to calculate sample size, the step-by-step process, how to keep tests safe for SEO, which tools to consider in 2026, and the statistical mistakes that ruin most experiments.
Key Takeaways
- An A/B test compares a control with a variation, randomly splitting traffic, to see which performs better on one primary metric.
- Start from evidence (analytics, heatmaps, user feedback) and write a clear hypothesis before building anything.
- Decide sample size and duration in advance; stopping as soon as results look significant inflates false positives.
- Follow Google's testing guidance (no cloaking, rel=canonical, 302 redirects, end tests promptly) to keep experiments SEO-safe.
- Record every test, including losers; the learning is as valuable as the wins.
What is A/B testing?
A/B testing (or split testing) is a controlled experiment. Visitors are randomly assigned to see either the current version (the control, A) or a changed version (the variation, B). Because assignment is random, the difference in results can be attributed to the change rather than to who happened to visit.
| Test type | What changes | When to use |
|---|---|---|
| A/B test | One version vs one alternative | Most situations; simplest to analyse |
| A/B/n test | Control vs several alternatives | When you have a few strong ideas and plenty of traffic |
| Split URL test | Entirely different pages on separate URLs | Redesigns or different page templates |
| Multivariate test | Several elements in combination | High-traffic sites studying interactions between elements |
| SEO split test | Groups of pages, measured on organic traffic | Testing changes for search; see SEO A/B testing |
This guide focuses on user-facing tests for conversion. SEO split tests work differently, because you test on groups of pages rather than groups of visitors.
What can you test?
- Landing pages: headlines, hero images, CTA wording and placement, form length, social proof, page layout. See our PPC landing pages guide.
- Product and pricing pages: price presentation, plan order, guarantees, delivery information.
- Checkout and forms: number of steps, guest checkout, field labels, error messages.
- Emails: subject lines, send times, content and offers, including abandoned cart emails.
- Ads: headlines, descriptions and creative; Google Ads has built-in experiments, and our guide to ad copywriting covers what to try.
- Lead capture: offers and form designs for lead magnets.
Write a hypothesis worth testing
A good hypothesis links evidence, a change and an expected outcome. Use this template:
Because [evidence], we believe that [change] for [audience]
will cause [outcome]. We will know this is true when
[primary metric] changes by [minimum effect].Example: "Because heatmaps show most mobile visitors never scroll to our quote form, we believe that adding a short form above the fold for mobile visitors will increase quote requests. We will know when form submission rate per visitor rises by at least 15%."
Evidence can come from GA4 funnels, heatmaps and session recordings, customer surveys, support tickets or sales calls. Tests based on evidence win more often than tests based on someone's opinion.
How much traffic do you need?
Sample size depends on three things: your baseline conversion rate, the smallest improvement you care about (minimum detectable effect), and how confident you want to be. Smaller effects and lower baseline rates need far more traffic.
Using a standard two-sided calculation with 95% confidence and 80% power, here are approximate visitors needed per variation:
| Baseline conversion rate | Target rate (relative lift) | Approx. visitors per variation |
|---|---|---|
| 3% | 4.5% (+50%) | ~2,500 |
| 3% | 3.6% (+20%) | ~13,900 |
| 3% | 3.3% (+10%) | ~53,000 |
| 10% | 12% (+20%) | ~3,800 |
These are illustrative figures from the standard formula; use a sample size calculator with your own numbers. If you cannot reach the required traffic in four to six weeks, test bolder changes, test higher up the funnel where volumes are larger, or use qualitative research instead.
Don't peek and stop early
The most common statistical mistake is checking results every day and stopping the test the moment it shows significance. Evan Miller's well-known analysis shows that repeated checking with a 5% significance threshold can produce a much higher false positive rate than 5%. His fix is straightforward:
“Committing to a sample size completely mitigates the problem described here.”
— Evan Miller, How Not To Run an A/B Test
If you need to stop early for business reasons, use a testing tool that supports sequential or Bayesian methods designed for continuous monitoring, and understand how it reports results.
How to run an A/B test: step by step
- Find the problem. Use analytics to locate pages or steps with high traffic and weak conversion.
- Research why. Review heatmaps, recordings and feedback.
- Write the hypothesis and choose one primary metric (for example, purchases per visitor) plus a few guardrail metrics (revenue per visitor, refund rate, page speed).
- Calculate sample size and duration. Run for at least one full week, ideally two, to cover weekday and weekend behaviour.
- Build and QA the variation on all major browsers and devices. Check tracking fires correctly in both versions.
- Launch with a random 50/50 split (unless you have a reason to do otherwise).
- Do not change anything mid-test: no new campaigns targeted at one version, no edits to the variation.
- Analyse at the planned end. Check the primary metric, confidence interval and guardrails. Segment results carefully, remembering that small segments are noisy.
- Decide and roll out. Implement the winner, or keep the control if results are flat or negative.
- Document the hypothesis, result and learning in a shared test log.
Keeping A/B tests SEO-safe
Google supports website testing but sets clear rules. Its documentation warns against cloaking:
“Don't show one set of URLs to Googlebot, and a different set to humans.”
— Google Search Central, A/B testing best practices for Search
- For split URL tests, add
rel="canonical"on variation URLs pointing to the original. - Use temporary (302) redirects, not permanent (301) ones, to send users to variations.
- End tests once you have enough data and remove test scripts and URLs promptly.
- Keep page speed in mind; heavy client-side testing scripts can cause flicker and slow pages. See our guide to Core Web Vitals.
A/B testing tools in 2026
Google Optimize, once the default free option, is gone; Google's help centre states that Optimize and Optimize 360 “are no longer available as of September 30, 2023.” Today's options include:
- Dedicated experimentation platforms such as Optimizely, VWO, AB Tasty and Convert.
- Feature-flag and product experimentation tools such as LaunchDarkly, Statsig and PostHog, popular with development teams.
- Built-in testing in email platforms, landing page builders and ad platforms (for example, Google Ads experiments).
- Server-side testing for sites that want to avoid flicker and keep full control.
Whatever you use, send experiment data into GA4 or your analytics platform so you can analyse results alongside your other metrics.
Common mistakes
- Stopping when it looks significant. Decide sample size in advance and stick to it.
- Testing with too little traffic. Tiny tests produce random "winners".
- Testing trivial changes. Button colour tests rarely move revenue; test meaningful changes to offer, message or flow.
- Too many metrics. Pick one primary metric; otherwise something will look significant by chance.
- Ignoring novelty effects. A new design can perform well briefly just because it is new; run full weeks.
- Not checking tracking. Broken tracking in one variation invalidates the test.
- Forgetting to log losers. Failed tests teach you what customers do not care about.
Related Guides
- Content Marketing KPIs: What to Measure
- Heatmaps: How to Use Them to Improve Your Website
- Customer Journey Mapping: A Practical Guide
Frequently Asked Questions
How long should an A/B test run?
Until it reaches the sample size you calculated in advance, and for at least one full week (ideally two) to cover different days. Avoid running far longer than needed.
What is statistical significance in A/B testing?
It indicates how unlikely your result would be if there were no real difference. A 95% confidence level is common, but it only holds if you do not stop the test early based on interim results.
Does A/B testing hurt SEO?
Not if you follow Google's guidance: no cloaking, use rel=canonical on variation URLs, use 302 redirects and end tests promptly.
What should I A/B test first?
High-traffic, high-value pages where you have evidence of a problem, such as a landing page with many visits but few enquiries.
Can small websites run A/B tests?
Yes, but they need bigger changes and patience. With low traffic, test bold variations or use qualitative methods like user testing and heatmaps.
Conclusion
A/B testing works when it is treated as an experiment, not a guessing game: start from evidence, write a clear hypothesis, size the test properly, let it run, and learn from every result. Over time, a disciplined testing programme compounds into significantly better pages, emails and ads.
Want help building a testing programme that grows conversions? See our guide to conversion rate optimization, explore our conversion optimization services, or request a free quote.
