ab-testing

Do you know that A/B testing can be used to help your business? From improving your copy to streamlining your design for the best possible user experience to removing friction from your checkout process, conversion rate optimization can improve the bottom line and reach your business growth and revenue goals.

One of the most common requests is for case studies. Entrepreneurs and business owners are looking for examples of “how it’s been done.” These inside looks at testing can be helpful at every step of the process, from identifying what needs to be tested to creating hypotheses, from designing your tests to implementing the results in your business. Below is a selection of case studies from around the web which help you to understand some specific aspects of the A/B testing process. Let’s begin.

AdX

I am going to share a simple price floor A/B test in AdX that resulted in a revenue increase of 10%. I’ll also show you the details of how the test was run, so you can give it a try for yourself to see if it boosts your website’s revenue.

The Setup

The test was confined to all ad units on desktop traffic in the US, which comprises the majority of the traffic and an even higher percentage of the revenue. Then the test groups from a/b testing framework was used to target different price floors to different users. For one group of users, the “A” channel, there was no price floor, which was the existing setup. For the other group of users, the “B” channel, the price floor was set at $1, a floor that was arrived at based on a more recent analysis using the price floor model. After that rules in AdX were created for each group to make reporting on the results easier. Here are what the rules looked like in AdX:

ab-testing-adx

The included tags ensured that the focus was on US, desktop inventory, by channel, for all those add units.

The Results

The test was run over a 24 hour period across more than 7 million ad requests. In AdX the “Price Rules” report  was used to analyze the results. Here’s what was found:

ab-testing-adx

Here you see the eCPM increased by 30% – a significant gain – while the matched coverage rate dropped by 28% – an even more significant drop. If those unmatched impressions went unserved, the net impact of the test would be a 6% decline in the original revenue. The impact of the test hinges on the recovery rate for those unmatched impressions.

A simple model was created to run a scenario analysis on the revenue impact given the full range of recovery rates. Here’s what was found:

ab-testing-adx

Unfortunately, AdX reporting doesn’t currently allow for the calculation of the actual recovery rate, so an assumption was made. At just a 30% recovery rate or above, the experiment had a net positive impact on revenue. They went with a relatively conservative 50-60% rate, on the idea that, on average, one of the header bidder partners would fill the impression with at least half of the revenue AdX would provide. Under this scenario, the result was an approximate 10% increase in revenue. Not bad.

Given the positive impact of the price floor across most recovery rate scenarios, the new floor was implemented immediately. But just to be sure the impact was sustained over time, they kept a portion of the traffic price-floor free, essentially continuing to test that the floor was in fact adding value.

Of course, there are questions that have to be explored in future experiments. Anyway, this experiment shows you how easy it is to run price floor A/B tests.

CloudSponge

CloudSponge is a service designed to make website owners’ lives easier by automating the day to day management and sharing of their customers’ address books. This Software as a Service product enables website owners to rapidly scale their business and seamlessly invite their members’ colleagues and friends to new services, without the hassles of having to develop their own systems from scratch.

The problem

Jay Gibb the founder of CloudSponge was unhappy with the conversion rate of the existing website and felt the company’s existing web design and logo was dated and needed to be refreshed. Jay decided to look into crowdsourcing as a solution to his company’s design problems.

ab-testing-cloudsponge

CloudSponge’s previous homepage design

After writing a brief and posting the project CloudSponge received a unique design for his website.

ab-testing-cloudsponge

Benefits

Within just 10 days this result was achieved at a fraction of the cost of using a regular design service, saving the company thousands of dollars in costs.

Since launching the new website design, CloudSponge has seen an increase in their conversion rate by over 33%.

Summary

After creating a new design CloudSponge managed to grow its business and improve its ROI on every marketing dollar spent. This was achieved on a budget of less than $1500 and in just 10 days.

A Hume

Country clothing retailer A Hume first realised the benefits of testing back in 2012, when it redesigned its checkout login screen and billing details process.

According to James Abbott, the brand’s director of digital strategy and optimisation, the results were impressive.

The company certainly did more, running a number of tests on product and category pages to see where it could make improvements. And all of this has had a positive impact on the bottom line.

Let’s have a look at three successful tests A Hume has carried out since last year.

Test 1: product page variation, July 2014

The first test revolved around product page imagery. A Hume wanted to test out moving the images to the left and then raising the selection process, call to action and messaging higher up the page.

They wanted to improve the user experience (UX) of the page by making the imagery the lead aspect, while making the call to action more visible on widescreen monitors and tablets.

You can see the original control page along with the updated version below.

The original version

ab-testing-a-hume

The updated version

ab-testing-a-hume

The results were great. Add-to-basket levels increased by 27.65%.

Test 2: category page variation, July 2015

A Hume’s tweed cap category page featured standalone product images, but they wanted to test whether or not the inclusion of lifestyle images with models wearing the products would improve click-through rates (CTR).

Below you can see a comparison between the original and updated pages.

The original page

ab-testing-a-hume

The updated page

ab-testing-a-hume

The results were again quite good. CTR increased by 15.9%.

Test 3: Product page variation, September 2015

A Hume reworked the copy on its footwear product pages to make it more readable and easy to scan. At the same time they tested out a different approach to imagery.

A Hume created both a short and long product description, made all the copy more concise and used headings and lists to improve the user experience.

For the images, the brand used stills from its HD videos to show boots stacked on top of each other to give a better overall view of the product.

You can see the before and after images for these tests below.

The original page with updated test page (text only)

ab-testing-a-human

The updated test page (text and imagery)

ab-testing-a-human

The results were the following. The text variation test resulted in an 8.21% increase in add-to-basket levels.

The second test including both text and image variation resulted in a 14.51% increase in add-to-basket levels.

The largest retailers in Turkey

One of the largest retailers in Turkey faced a problem that their page was not properly presenting the value preposition. There was a high bounce around the fact: there was a higher than normal pre-product page abandonment rate.

So, let’s move to this case study.

The hypothesis is:

By emphasizing the value of the company to the visitors on the category pages you’ll increase flow to product pages as well as increase conversion.

That was the hypothesis for this particular case study.

So, when we look at the control we can see the left navigation. There’s also the grid view of the products. There’s a top advertisement, carousel of “top selling products”.

Variation 1:
  • We removed the top ad.
  • We placed a new banner, which says “7 day-money-back guaranty”
  • And we removed the carousel

ab-testing

Variation 2:
  • We also removed top ad
  • We placed a new banner “customer satisfaction guaranteed or Your money back”
  • And we removed the carousel

ab-testing

Variation 3:
  • We removed the top ad
  • We placed a new banner “lowest prices, reliable service, speedy delivery”
  • And we also removed the carousel.

ab-testing

So we are going to have a side by side look here

ab-testing

And what we are going to do is to ask the audience (visitors) which design generated the most sales.

We usually lounge upon with these particular case studies to see what you would think and which design you actually would chose.

Finally the winner is the variation 2 and it has increased conversion by 66,8%

Here’re the results, you can see variation 2, variation 1 and variation 3. They all actually produce some uplifts.

ab-testing

Design variation 2 as you see over here has 66,8% increase and 93,1% confidence rate are really healthy uplifts.

And now we are going to look at categories pages

  • Of all pages on an ecommerce site, we find the value preposition is least present on category pages, which is really interesting.
  • Category pages also in general lack “personality”
  • And what they should do in the end is just funnel the visitors to the product pages. However many categories pages do not do that.

Now, let’s discuss the keys to successful category pages:

Key number 1 is to utilize prime real-estate on the page to increase Buyer Momentum through:

  • Inserting effective, persuasive banner messaging, which resonates with different visitor personas
  • Avoiding the use of multiple auxiliary navigations

When we look at the control again what we see on it is that:

  • The product grid is pushed lower on the page because of the carousel and the top banner
  • The top banner is not targeted to all users (and may not be a relevant product)
  • Prime real-estate on page is not used effectively

Variation 2, the winner:

  • Utilizing prime real-estate by moving product grid up. The prime product was moved significantly up.
  • Using the banner space to emphasize the value.
  • Remove elements that although offer additional navigational paths, in this case were obstructing scent.

And again there are keys to successful category pages.

Key number 2 is to provide sufficient scent through multiple navigational paths to accommodate visitor needs

  • Remove elements that reduce scent
  • Test before placing additional navigational paths and consider your visitor’s cognitive progression
  • Ensure that “left navigation” or “refine search” features meet visitor expectations by presenting the most pertinent drill down features first.

Considering any, for example, website a lot of time they have lots of different products, lots of different options, you want to make sure what you test, which options are most important. Is it a break down by colour, size. Whatever you have, you have to be sure that you consider all these factors.

Conclusion

I’m sure you’ll agree that while there is a room for improvement, none of the original pages displayed above were particularly offensive.

But as you can see from the results shown here, even the smallest of changes can have a big impact on the way people engage with a site.

Every ecommerce site should be constantly testing its pages to see whether they can get a better response from visitors, even on pages that are already performing well.