A/B Testing
Comparing two versions to see which performs better with real users.
A/B testing (or split testing) shows different versions of a page or feature to different users and measures which drives the better outcome. It replaces opinion with data for high-traffic decisions.
It's most useful when you have enough traffic for statistical significance; below that, qualitative research and best practices are more reliable guides.
Key characteristics
- Compares two or more variants by splitting live traffic and measuring a goal metric.
- Needs a hypothesis, a primary metric and enough sample size for statistical significance.
- Removes opinion from decisions by measuring real behaviour.
- Multivariate testing extends it to several elements at once.
Example
A team tests a green vs blue CTA button; after enough traffic, the green variant converts 8% better with significance, so it ships.
Frequently asked questions
How long should an A/B test run?
Until it reaches a pre-calculated sample size and statistical significance, usually at least one full business cycle (often 1–2+ weeks) to avoid day-of-week and novelty effects.
What is statistical significance in A/B testing?
It's the confidence that an observed difference isn't due to chance — commonly a 95% confidence level. Calling a winner before reaching it risks acting on noise.
Related terms
Need this done, not just defined?
I design and build it — Framer sites, product UI/UX, and more.