Over the coming months, we’ll release several posts covering the most basic aspects of testing – which is the purpose of this post – and experimentation, through to detailed explanations covering everything from how to generate test ideas through to building a test, analysing the results and driving site updates.
A beginner’s guide to A/B Testing for marketers, product managers and everyone in between
If you’ve ever wondered why some websites convert visitors better than others, or how top-performing companies constantly improve their digital experiences, the answer often lies behind website A/B testing. (For the purposes of this post, assume website covers both a website and an app.)
Whether you’re in marketing, product or leadership, understanding the purpose and benefits of testing is a powerful step toward making better, data-driven decisions.
In this post, we’ll break down what A/B testing is, why it matters and how it works, whilst avoiding as much jargon as possible.
What Is A/B Testing?
A/B testing refers to a method of comparing two or more versions of something, like a webpage, email or app screen in order to learn which one performs better.
At its most basic level, you would show Version A to half your users and Version B to the other half. Then measure which version gets the most clicks, signups, purchases or whatever your measurable goal is.
Why should I Use A/B testing?
In a world of opinions, A/B testing gives you proof about which version is best.
Here’s why it’s powerful:
- It replaces guesswork with data and facts
- It helps you understand what actually works in a real-world scenario
- It minimises risk when making changes – you don’t want to promote a poorly performing change
- It creates a culture of continuous improvement. It can deliver big gains but it’s also great when looking for ongoing marginal gains
Ultimately, whether you’re tweaking headlines or launching new pricing, testing helps you move forward with confidence, not just intuition.
A/B Testing Examples
Here are some simple examples of A/B tests. But simple can still have a huge impact.
Headline Test:
A: “Start Your Free Trial” vs B: “Try It Free for 14 Days”
CTA Button Colour:
A: Green vs B: Orange
Email Subject Line:
A: “Your Cart Is Waiting” vs B: “Don’t Miss Out on Your Items!”
Pricing Page Layout:
A: in order of Bronze, Silver and Gold vs B: Silver (most popular) option first
Common Misconceptions
Let’s dispel a few myths:
“A/B testing is just for big companies.”
Not at all. You can start small, even with 1,000 visits a week. Instead, think about is as structured learning which is not limited to huge volumes.
“It’s only about button colours.”
A button test is a very simple test. Whereas, more effective tests involve value propositions, user flows, pricing, copy etc. All core components of any digital product or service.
“One test will solve everything, so I only need to test once.”
Think of A/B testing as a habit, not a one-off event. It’s about building momentum over time to continually improve conversion.
How to Start Testing?
Starting simple is the best approach, KISS (“keep it simple, stupid”). You’ll need:
- A goal: What are you trying to improve, eg more signups?
- Traffic: Enough users to detect statistical differences (even 1,000 or less a month can work)
- A tool: Platforms like, Adobe Target, Monetate, Optimizely, VWO, Convert etc
- A plan: Decide what to test, how long to run it and how you’ll measure success
- Resource: You’ll need to build the test and QA it before turning the test on. Then you’ll need to review the results and decide what to do with the results
TL;DR
A/B testing is a way to compare two versions of something to see which performs better.
It helps you make better decisions using real data and statistical analysis, not just gut feelings.
You don’t need to be a data scientist or a large business to start.
The best A/B tests become part of a wider optimisation strategy, which we can help build and define with you.
Need Help Getting Started?
If you’re curious about how A/B testing could work for your business but you’re not sure where to begin, get in touch with us and we’ll give you our advice and suggestions.
Next time …
“Why Most Tests Fail – and What to Do Instead”
Follow us on LinkedIn and subscribe to our newsletter to be notified when it drops. And you can read other posts on a wider range of topics here.
Test smarter, faster and with more impact.
