A/B Testing

Playbook
Table of contents
Welcome to the world of web experimentation 3
A/B testing types 5

Choosing an A/B testing tool 7

7 popular tools compared 9

Deciding what to test: a practical guide 11

Forming your hypothesis 12
PIE prioritization framework 13
Common A/B testing goals 4

3-step plan for successful tests 18

Step 1: preliminary research and evidence-based hypothesizing 19
Step 2: pre-test planning and experiment set-up 20
Planning template 22
Step 3: thorough analysis of all versions 23
Results reporting template 24

Conclusion 25

About Contentsquare 26

A/B Testing Playbook 2

Introduction For any team with conversion-oriented KPIs, every decision made has a quantifiable impact
on business outcomes. Gut feeling is a good starting point for making decisions on websites

Welcome and products, but it’s just one piece of a very fragile puzzle: budgets are often tight, users are
unpredictable, and changes are risky.

to the world A/B testing is a user experience (UX) research and web experimentation methodology that
proves your hypothesis right or wrong. Running a test enables you to launch variations of web
pages, apps, or products that you know will yield the best results for a given conversion goal—

of web no guesswork necessary.

But, the biggest A/B testing myth is that the benefits end there. Hidden in the losing version

experimentation of your test are revenue-boosting insights capable of informing future tests and adding real
business value—you just need to know how to look for them.

That’s why we’ve created this A/B testing tool kit. We’re here to help you mine every nugget of
insight from your test so you can give your audience a digital experience that’s easy, intuitive,
and exactly what they need.

At its most basic level, A/B testing allows product, UX, marketing, and ecommerce teams to
compare two versions of a web page, app, product, or feature and then determine the top-
performing candidate. During the testing period, these two versions—called a control and
variation—launch simultaneously to different audiences. Statistical analysis determines the
winning version. The version that yields the best result for a given conversion goal is then rolled
out to the entire audience.

A/B Testing Playbook 3

An A/B test in action
British beauty products Space NK used Variation 1 left the stars black and showed an
average rating and number of reviews, while
Heatmaps to discover that customers who
variation 2 changed the color of the stars and
clicked on product reviews were more likely provided a clearer, contextual CTA.
to add products to their checkout (28.8%)
compared to those who didn’t (12.6%). Yet on
average, only 6.88% of product page visitors Variation 1 (Control)

+30%
engaged with the review stars. Space NK Results:
hypothesized that getting more customers to
read reviews would increase conversion and
revenue. increased
overall conversions
They tested two variations of the product
review stars and CTA, emphasizing the color
Variation 2
and style. They left variation 1 as the control
and to variation 2, they changed the color
of the stars and updated the CTA copy. We
found that variation 2 encouraged more
visitors to click on review stars and interact
with customer reviews. This ultimately
increased conversion by +30%.

A/B Testing Playbook Introduction 4

6 types of 1. A/B test 2. Split URL test 3. Multivariate test

experimentation
• Usually tests 2 versions • Compares 2 versions of • Tests numerous
While A/B testing is the most common of a web page, app, a web page, but each combinations of variables
example of experimentation, there are several product, or feature is hosted on a different at once, so may include
additional testing methodologies you can use hosted on the same URL URL, and traffic is equally dozens of versions of a
to get the data you need, depending on your distributed between them web page
• Lets you answer targeted
role, industry, and business size.
questions about highly • Recommended for bigger • Used to determine which
specific modifications tests that require more combination of elements
Here are 6 of the most useful experimentation significant design or produces the most
types. Which ones are non-negotiable for backend changes conversions
your experimentation program? Once you
Example
know the answer, you’ll be better positioned
to choose a testing tool. Comparing a blue CTA
Example Example
button to an orange one to
see which gets more clicks Comparing 2 vastly Comparing 2 image options
different landing page and 4 CTA colors in 8
designs that offer different different combinations
page experiences

A/B Testing Playbook Introduction 5

4. Multi-page or multi-funnel test 5. Server-side test 6. Mobile app testing

• An extensive test that looks at users’ journeys • Tests high-impact non-user interface (UI) variations, • A server-side test that compares different
through multiple pages that are all part of a like different algorithms, architectures, and other versions of an in-app experience on mobile
particular funnel backend variables.
• Used to improve overall mobile app
• Used to optimize the design of several pages • Used by developers to go beyond visual changes and experience and encourage users to convert.
to encourage bottom-of-funnel conversions compare complex product variables, rendering tests
on the server instead of the user’s browser

• In contrast to client-side testing, which is any type of Example

Example testing occurring in the user’s browser that requires no Comparing different in-app messaging
Comparing 2 different product names on an coding. variations to determine which is more effective
ecommerce listing and then carrying both in nudging users to perform a desired action
names through to the corresponding checkout
page variations to ensure a consistent UX
Example
Comparing 2 vastly different landing page designs that
offer different page experiences

A/B Testing Playbook Introduction 6

Section 1

Choosing
an A/B testing tool

A/B Testing Playbook 7

Choosing an A/B ⬜ Supports different types of tests ⬜ Is easy and intuitive to use

testing tool Consider features like multivariate

testing, split URL testing, and mobile
Running A/B tests shouldn’t be too labor-
intensive. Choose a tool that prioritizes
app testing. Even if you’re in the early user-friendliness to make testing
Choosing the right tool is a critical part of stages of your experimentation program, more efficient. Look for features like
your testing strategy, and there are a few access to these diverse testing methods customizable dashboards, flexible editors,
important factors to consider. We recommend will make it easy to tailor experiments to an up-to-date library of resources, and
looking for a tool that checks most of these keep up with the increasing complexity excellent customer support.
boxes: of your business needs.

⬜ Provides advanced targeting options

⬜ Integrates with other tools To get impactful results from your test, you
Your A/B testing tool shouldn’t exist in need the ability to target highly specific
isolation. If you want to get the most cohorts of users. Find a tool that can target
meaningful insights from your results, it people based on conditions like location,
should seamlessly form part of a holistic time, device, and traffic source. Different
tech stack that allows you to enrich your audiences provide different insights, and
A/B test data and distill the maximum targeting the most important group for
amount of insight. each test ensures more powerful results.

⬜ Gives accurate results

The best tools are transparent about how they run tests and analyze results. Choose a tool that
considers sample size when determining statistical significance, employs proper randomization
techniques to minimize bias, and gives you data free from anomalies and inconsistencies.

A/B Testing Playbook Choosing an A/B testing tool 8

Popular tools compared AB Tasty ClickFunnels Kameleoon

Here are 7 popular tools, and they’re a good

Who it’s for Who it’s for Who it’s for
place to start if you’re shopping around. All
Marketing, UX, and Marketing and ecommerce teams Marketing, product, ecommerce,
of them integrate with behavior analytics ecommerce teams and engineering teams
Best for SMBs
tools like heatmaps, session recordings, and Enterprise-grade Enterprise-grade
surveys (or have them built-in). This means Tests supported
Tests supported A/B testing (specifically for Tests supported
you can squeeze additional qualitative data A/B, split URL, multivariate, ClickFunnels campaign pages) A/B, split URL, multivariate,
from your winning and losing variants for multi-page/multi-funnel, mobile app, server-side
mobile app, client-side, and Pros
better insight into why the results look the
server-side Easy to use for conversion rate Pros
way they do. optimization (CRO) beginners Unified platform for client- and
Pros server-side testing, enabling
Feature-rich and highly
Unlimited experimentation cross-functional collaboration
A number of companies listed belong to our customizable
Easy to use for teams with Excellent support
partner ecosystem, which is designed to limited technical skills Cons
enhance our platfoem and deliver benefits Support can be slow to respond Cons
Cons Steep learning curve compared
to our customers. This ecosystem includes Slower page load speeds
Fewer advanced to some other tools
certified Technology Partners, who provide experimentation capabilities Can be complicated for less
Are they in the Contentsquare
products and technologies that integrate with Slower page load speeds partner ecosystem? technical users
and extend the Contentsquare platform. Are they in the Contentsquare
No
Are they in the Contentsquare
partner ecosystem? partner ecosystem?
G2* rating
Technology partner Technology partner
4.6

G2* rating G2* rating

4.5 4.7

A/B Testing Playbook Choosing an A/B testing tool 9

Omniconvert Optimizely Unbounce Convert

Who it’s for Who it’s for Who it’s for Who it’s for
Product, marketing, engineering, and Marketing, UX, and ecommerce teams Marketing teams Marketing, UX, and ecommerce
ecommerce teams Enterprise-grade Best for SMBs teams
Enterprise-grade Enterprise-grade
Tests supported Tests supported
Tests supported A/B, split URL, multivariate, A/B and multivariate tests (specifically for Tests supported
A/B, split URL multi-page/multi-funnel, and mobile app Unbounce landing pages) A/B, split URL, multivariate, multi-
page
Pros Pros Pros
Powerful segmentation engine A robust customer data platform for Integrates with many popular marketing tools Pros
Includes surveys for additional in-depth insights Automatically directs visitors to the Full-stack capabilities included in
qualitative insights Clear and concise results dashboard best-performing variant based on real-time data every plan
Intuitive drag-and-drop system
Cons Cons Cons
Can be confusing for less technical users Steep learning curve compared to some Limited access to features in lower plans Cons
Complex user interface compared to other tools Can’t split test existing landing pages created Has some interface usability issues
other tools On the pricier side outside of Unbounce Onboarding is limited

Are they in the Contentsquare partner Are they in the Contentsquare partner Are they in the Contentsquare partner Are they in the Contentsquare
ecosystem? ecosystem? ecosystem? partner ecosystem?
No Technology Partner No Technology partner

G2* rating G2* rating G2* rating G2* rating

4.5 4.6 4.4 4.7

*G2 is a tech marketplace that compares software and services based on user ratings and social
data. The G2 rating is a standardized score used to compare products within the same category.

A/B Testing Playbook Choosing an A/B testing tool 10

Section 2

Deciding what to test:

a practical guide
While it’s impossible to rely on gut feeling
alone, it will often be the launchpad for your
experiment hypothesis. A hypothesis is an
educated, testable statement that should

• Propose a solution or explanation to a problem

• Predict an outcome for the experiment
• Provide reasons for the expected result

Given 20 minutes, you could probably come

up with 100 ideas about changes to your site
or product that could impact your users—but
that doesn’t mean they are all worth testing.
The ones you investigate further usually involve
additional research or digging deeper into
existing data.

A/B Testing Playbook 11

Forming your hypothesis

Most teams will formulate a hypothesis

based on blended quantitative and qualitative Based on [existing research and results], we believe that [implementing X change]
insights. The result is a statement that would will result in [desired outcome].
look something like this:

For example, the Space NK team’s hypothesis Based on insights from Contentsquare’s analysis on product reviews, we believe
for their customer review stars test we that getting more customers to read reviews will result in increased conversion
mentioned earlier could look like this: and revenue.

An evidence-based hypothesis that follows this format ensures you don’t waste time
going down rabbit holes that ultimately lead nowhere. So, instead of immediately
launching into a test, spend some time doing the research into the business metrics
that need improvement. Dig into past initiatives that may have impacted these numbers
and look at existing behavior analytics data. Seeing what users currently do and how
they feel will help inform your A/B test.

A/B Testing Playbook Deciding what to test: a practical guide 12

Prioritize like a pro with
this simple framework
Prioritization is a crucial aspect of any Potential test page Potential Importance Ease Priority ranking
A/B testing program—resources are finite
and tests require time to set up. If you
Homepage 10 10 3 7.7
find yourself overwhelmed with ideas and
pages to test, using a simple prioritization
framework will help guarantee a good return
on investment (ROI) for your efforts. Luckily,
prioritization is as easy as PIE.

The popular PIE framework looks at 3 criteria:

potential (how much improvement can be
made), importance (how valuable this page
or experience is), and ease (how easy it is to
test). Then, you score each page from 1 (low)
to 10 (high) based on these criteria and divide
by 3 to get your priority ranking..

Use this simple matrix to create an objective

test priority list.

A/B Testing Playbook Deciding what to test: a practical guide 13

Case study

How Mitre 10 increased

conversion rates using
A/B testing
Home improvement chain Mitre 10 discovered
via our Journey Analysis capability that
users who landed on a “no results” page
after inputting a search query would quickly
bounce.

They decided to A/B test whether adding a

product carousel on the previously empty
‘no results’ search page would keep users on

+42.3%
their site. Mitre 10 wanted to A/B test this “no results” Results:
page to prevent users from quickly bouncing
off the site.
The variation resulted in a +42.3% uplift in
revenue for the search audience segment. uplift
in revenue for the search
audience segment

A/B Testing Playbook Deciding what to test: a practical guide 14

6 A/B testing goals 1. Increase conversion rate 2. Increase click-through rate

and common ways

to impact them Abandonment rate refers to the percentage
of tasks users start but don’t complete,
Bounce rate—the percentage of visitors
entering and quickly leaving your website
like leaving a survey midway or adding an without taking additional action—is a good
While every business is different, there item to a shopping cart but not purchasing. indicator of visitor interest. A high bounce
are a few universal metrics and goals that A/B testing to prevent abandonment will rate is often indicative of website design
marketing, product, UX, and ecommerce ultimately increase revenue for any business issues or a content mismatch, giving you
teams track. Considering these goals—and selling online. more insight into the effectiveness of your
the changes that might influence them—is experiment.

a good place to start if you’re in the early

stages of your experimentation program. Changes to test
Changes to test
• Replacing product imagery on listings
• Reducing checkout steps to simplify UX • Adjusting the messaging and placement of
• Comparing a multi- and single-page website copy that communicates the value
checkout experience of your website
• Improving page load speed to avoid user
frustration
• Fully redesigning a page to improve its
experience

A/B Testing Playbook Deciding what to test: a practical guide 15

3. Decrease 4. Decrease bounce rate 5. Increase retention rate 6. Increase exposure rate
abandonment rate

Bounce rate—the percentage Retention rate is the percentage of Exposure rate in Heatmaps shows how
Abandonment rate refers to the of visitors entering and quickly users revisiting a website or specific far down a page users scroll.
percentage of tasks users start leaving your website without page after a certain period and is a
but don’t complete, like leaving taking additional action—is a valuable sign of customer loyalty. It’s is a key metric to track when A/B
a survey midway or adding an good indicator of visitor interest. Comparing retention rates between testing, as it can help you to better
item to a shopping cart but not A high bounce rate is often different A/B test variations helps understand how much users scroll,
purchasing. A/B testing to prevent indicative of website design you understand what encourages especially to elements you want them
abandonment will ultimately issues or a content mismatch, users to return and engage with your to engage with. With this information,
increase revenue for any business giving you more insight into the website or product. you can make data-driven empirical
selling online. effectiveness of your experiment. sizing and placement adjustments.

Changes to test
Changes to test Changes to test Changes to test
• Experimenting with in-app
• Replacing product imagery on • Adjusting the messaging and messaging to guide users to • Redesigning the content hierarchy to
listings placement of website copy valuable product features compare average time on page
• Reducing checkout steps to that communicates the value of • Trying out different onboarding • Placing important information above
simplify UX your website flows to improve users’ the fold so users get important
• Comparing a multi- and single- • Improving page load speed to understanding of a product from information, faster
page checkout experience avoid user frustration the get-go • Experimenting with design elements
• Fully redesigning a page to • Testing loyalty programs, referral like headings, colors, and image
improve its experience programs, or incentive offers to placement
encourage repeat purchases

A/B Testing Playbook Deciding what to test: a practical guide 16

Case study
Discovery Hypothesis
How AVON increased When Avon’s digital team analyzed their The AVON team hypothesized that reducing
revenue by +35% makeup category page, they found the
product carousel had a low exposure rate on
the height of the banner and bringing the
product carousel higher up on the page would
desktop—40% of customers were not even improve exposure and conversion rate.
Beauty company AVON used our Heatmaps seeing it. Visitors were viewing the banner
to get a visual overview of their makeup image at the top of the page but weren’t
category page to see what their users viewed scrolling down to the product carousel, which

+35%
and engaged with the most. was located beneath the fold. Results:

Assumption
increase
in overall revenue attributed
Based on this evidence, the team made an to the zone
educated guess that the banner size and page
structure could be reorganized.

Result

+44%
Results:
Running an A/B test with a shorter banner
resulted in a +44% increase to the exposure
increase rate (from 57% in the control to 82% in the
variant). The team also noticed a +24%
to the exposure rate
increase in click rate and overall revenue
attributed to the zone increased by +35%.

A/B Testing Playbook Deciding what to test: a practical guide 17

Section 3

Testing, 1, 2, 3:
a 3-step plan
for successful
tests

A/B Testing Playbook 18

Step 1

Preliminary research
Case study

Creating a winning hypothesis based on user engagement

and evidence-based
hypothesizing U.K. retail bank NatWest used our
Experience Analytics platform to analyze
Analyze existing data and customer feedback its mobile app savings hub page. They
to identify potential areas for improvement. found the page had a high exit rate, with With this insight, the team
Once these insights are gathered, formulate users scrolling far down the page before hypothesized that the hub
a hypothesis that clearly states the outcome leaving. Since high volumes of users came page design didn’t have enough
you expect from the proposed changes. to NatWest through this page, it was key information that users were
important to remedy the problem quickly. looking for.

This hypothesis should be rooted in data and

insights gathered during your research phase.
Use it to guide the direction of the A/B test
and ensure that each variant serves a specific
They A/B tested a variant Fixed Rate ISA card on the page, which featured scannable
purpose in addressing the identified issues or and more digestible content. Using our side-by-side analysis of both versions, they
goals. found this change resulted in a significant uplift in user-to-application completion
rate. By performing this test, analyzing it, and then implementing the changes,
NatWest was able to achieve a very powerful customer journey optimization.

Read the full NatWest case study

A/B Testing Playbook 3-step plan for successful tests 19

Step 2
Decide on a single primary metric
Pre-test planning you want to measure
and experiment set-up You can (and should) have secondary ones—especially
for bigger tests running over longer periods—but it’s
You’ve chosen your tool, conducted some important to determine the main metric you’ll focus on.
preliminary research into the metrics that
matter to your business, and developed your
hypothesis. Now, you’re ready to set up your
Create 2 versions of the element you’re testing
experiment. The setup process will differ
depending on your tool, but here are a few This might be an email subject line, CTA button, landing
things you’ll need to do: page layout—anything you believe could impact
conversions. The baseline or control version (A) displays
the current element, design, or page, while the variation
(B) deploys the change or group of changes you want to
study. Both versions should be identical in all other ways.

Take a subset of your users or members

to act as your sample

Split your sample evenly into 2 groups using random

assignment. One group sees the control version, and the
other encounters the variation.

A/B Testing Playbook 3-step plan for successful tests 20

Case study

See it in action
This led the team to simplify the page with a smaller hero
image, remove the category tiles, and use only product tiles,
to help boost product visibility.
U.K. online grocery business
Ocado Retail conducted a
Heatmap analysis in our live
browser extension on their
Christmas promotional page.
The team found that 0.76% of
Desktop click rate
desktop users and 1.24% of
mobile users were clicking on
the unclickable
hero image and that scroll
Variation 1 (Control) Variation 2
depth was low. A zoning analysis of Ocado’s
Christmas Wondermarket page,
showing click rate on desktop (L) Ocado’s A/B test with (control) and without (variation 2)
and mobile (R) devices. Mobile click rate the hero banner.

They hypothesized that by removing the hero image, the product tiles would sit above the fold, making it easier for their
customers to find the products they were looking for. They then ran an A/B test with a variation of the page without the
hero banner. This variation saw no impact on order conversion, but it did have a positive uplift on their secondary metrics.

A/B Testing Playbook 3-step plan for successful tests 21

Decide what your confidence level should be

A confidence level determines how sure you can be of your results. As the researcher, this
value is totally up to you. In market research, it’s common to use a 95% confidence level.
This means if you ran the experiment 20 times, you’d get the same results (with a margin of
error) about 19 times. A higher confidence level provides greater certainty but might require
a larger sample size to achieve, while a lower confidence level may be more practical but
comes with the risk of drawing inaccurate conclusions.

Determine statistical power Determine the sample size

and timeframe for your test Tip
This refers to the probability that
the test will correctly identify a true This process involves balancing statistical
Use our free planning
difference or effect, if one exists. A considerations with practical constraints template
properly powered test is more likely such as time, cost, and resources. If
to detect differences, but if your test you’re new to testing and need help We just listed a whole lot of
is underpowered, you’re at high risk deciding on the right sample size, try information to keep track
of failing to reject a false null. Your an online sample size calculator. Once of! Make a copy of our A/B
A/B testing tool should calculate the you’ve decided on the right size, you testing planning template—
statistical power for you based on should easily be able to set it in your A/B the same one we use for
factors including sample size and testing tool and start running the test. our experiments—and fill in
confidence level. each field before launching
your next experiment.

Use our template

A/B Testing Playbook 3-step plan for successful tests 22

Step 3
We recommend combining A/B
Thorough analysis testing with behavior analytics
tools like zone-based heatmaps,
of all versions replays, exit surveys, and interviews
to understand not just which
variation wins or loses, but why.
Once you become more familiar with A/B
This knowledge will help inform
testing, you’ll quickly learn that not every future successful tests and ensure
experiment will go the way you expect. you don’t waste time inadvertently
Similarly, some tests might have a runaway doing the same thing and expecting
winner, while others won’t be so cut and different results. For example,
dry. Analyzing both versions ensures you repeatedly running tests to change
the copy on a CTA won’t make sense
squeeze every last insight from your A/B
if users aren’t scrolling far enough to
test, even if the results aren’t what you see the CTA in the first place. Compare user behavior side-by-side on experiment control and
hoped for. variation pages using our Experience Analytics solution.
By adding a qualitative layer to
quantitative A/B testing data, you’ll
get a well-rounded understanding of
an experiment’s results that you can Some A/B testing tools, such as VWO and Omniconvert,
use to inform future tests. have behavior analytics built in. Many others integrate
with digital experience analytics platforms like ours for a
robust suite of digital experience and behavior analytics
tools that paint the full picture of user behavior on both
your control and variant pages. This capability isn’t
just limited to websites—our platform can also capture
insights from your mobile app tests.

A/B Testing Playbook 3-step plan for successful tests 23

What if neither variation is
statistically better?
Mark the test as inconclusive, indicating that the variable
you tested didn’t significantly impact the results—but
don’t stop there. Learn from the experience by using “When coming up with solutions to problems, I encourage my
the failed data to help you create a fresh iteration for a team to use the Triple Diamond Framework which consists of
new test. Take a closer look at the data for context—
3 steps: problem discovery, solution discovery and prototype or
the answer to why customers prefer one variation over
another—and determine if any user subgroups respond go-live. Contentsquare is crucial in all phases as it allows us to
differently to the variations really understand the problem, hypothesize potential solutions and
thoroughly analyze user behavior across testing variations with
greater depth than ever before. Plus, I believe it has significantly
enhanced our learning process regarding experiment variations in
user behavior, making it ten times better for us.”
Tip

Use our free results report template

Joshua Kreuger
Track your wins and learnings with our A/B test Manager Experimentation, Subscription Sales
results report template to share with stakeholders DPG Media
and get buy-in for follow-up experiments.

Use our template

A/B Testing Playbook 3-step plan for successful tests 24

Conclusion
As with all things CRO, A/B testing isn’t just Your product analytics tech
a process of finding a winning variation,
it’s about gathering actionable insights you
can apply to your website to improve your
stack needs Contentsquare
customer experience. Start using Contentsquare for free
in minutes and automatically capture data that
Remember that inconclusive and ‘failed’ tests helps you build better products.
give you a clearer idea of what doesn’t work
Book your demo
for your users. There are no losses in A/B
testing, only learnings—so gather your data
and get experimenting.

A/B Testing Playbook Conclusion 25

About
Contentsquare
Contentsquare is the all-in-one experience intelligence
platform designed to be easily used by anyone that
cares about digital journeys. With our flexible and
scalable platform, you quickly get a deep understanding
of your customers’ whole online journey. Our AI-powered
insights provide those “ah ha” moments you need to
deliver the right experiences. You get to work faster and
smarter with the confidence to know what to do next
to improve your digital experiences. Leading brands
use Contentsquare to grow their business, deliver more
customer happiness and move with greater agility in
a constantly changing world. Our insights are used to
optimize the experience on over 1.3 million websites
worldwide.

For more information, visit: contentsquare.com

A/B Testing Playbook About Contentsquare 26