Untitled design

A/B testing of a website is a relatively new, but a rather effective tool for increasing your website’s conversion rate. In Google Trends, it’s popularity is growing by 50-90 % per month. This evokes numerous questions which are difficult to answer. Which used to be difficult to answer, to be correct.

1. What Is A/B Testing?

This is an effective method for increasing your website’s  conversion rate. A/B testing allows to determine which variant of a website page is better at converting visitors to buyers.

By using A/B testing, you can see how changing a page design or some element affects this page’s conversion rate. A/B testing  allows to change your website only for the better: every decision is based on statistical data collected in an experiment. Moreover, it allows your team to avoid taking guesses or facing conflicts about the website, when each member of the team tries to force his/her opinion on others.


A/B testing is figures only. Figures based on visitors’ actions. Results of experiments speak for themselves and show you the right way. The final goal of A/B testing  is to increase your profits, number of registrations and downloads, etc. This goal is achieved through complete understanding of your audience.

2. How To Perform A/B Testing Of A Website?

You can perform  A/B testing by using Google Analytics Content Experiment or self-written scripts. To create an experiment via Google Analytics, go to Behavior > Experiments.




To read more about self-written scripts for web traffic distribution, visit phpabtest.com or blog.alexmaccaw.com/abba .  I have never used this option, though, and I’m not going to.

The main disadvantage of those two options is that you’ll have to create test variants of pages yourself.

In fact, you’ll have to create a new page with a certain change. A link to a new page will look like this: www.changeagain.me/variation=1. Another drawback of free options is lack of additional functions provided by special services.

Making experiments by using special services is much more convenient and time-efficient:

  • Changeagain.me (Our service for those who use Google Analytics with one click integration and completely unique pricing model – pay for experiments, not for impressions)
  • Optimizely.com (The largest A/B testing company focused on high-profile clients)
  • VWO.com (A major player in A/B testing market with a wide range of functions)
  • Convert.com

A full list of A/B testing services with description of functions and fees can be found here.

Special services offer the following advantages:

  • Creating test pages via visual editor is simple and does not require special coding skills
  • Additional functions
  • One-time code installation (experiments are updated automatically)

The major advantage of A/B testing services is a visual editor designed to create test versions of pages.

Developers from some medium/large companies (Amazon.com, Quora.com, Facebook.com, etc ) can write software to meet their individual needs. This is useful if  a range of functions provided by special services is not enough. It’s hard to say how much it costs, though.  You should estimate usefulness of your own development in each individual case.

3. A/B Testing: Where To Start?

The elements, that affect a conversion rate directly, should be tested first. Those include СТА buttons, forms, text elements on a landing page, images, and video clips. Each of those elements plays an important role in the sales funnel.

Upon landing on a page, a visitor sees  mostly texts (headings, sub-headings, descriptions), images, and video clips related to a product. This is where a visitor’s acquaintance with a product begins. By changing those elements, you can influence a visitor’s perception and first impressions of a product.

Then a visitor is invited to fill out a form and share his/her phone number, email, personal information, etc. This is a stage where a large number of potential clients go away. Therefore, form optimization can improve your website’s conversion rate drastically.

After completing a form, a visitor needs to send it, and this is where CTA (call-to-action) buttons enter the scene. CTA button clickability may depend on its message, color,  placement, size, etc.

As all those elements play an important role in interaction between a visitor and a website, they should be the first to be tested. Creating such experiments is rather easy. Moreover, they have a much higher probability of delivering  a positive/negative result.

4. How Many Changes Can I Make On a Test Page?

One of the rules of A/B testing is: 1 test – 1 hypothesis.

Here is an example of a hypothesis: “If we remove the field “Name” from the subscription form, more visitors will fill out the form, which will result in a larger number of subscribers.” A proper  A/B test includes one change only. When analyzing the results of your experiment, you will know exactly that your conversion rate was affected precisely by that change.

If alongside with the field “Name”, we change the message and color of the button “Subscribe” and the text inviting to complete a form, we will hardly know which change affected a conversion rate. It may have increased through the explanation text, while the changed button affected a conversion rate in a negative way. But the first effect overweighed  the second one. Or not? There’s no telling.

5. How Long Should I Run A Test To Get Significant Results?

Minimum duration of an A/B  test is 7-14 days

A week’s duration has been chosen for a reason: in such a way, you will see the results for each day of the week. You need to wait for 7 days, because visitors may behave differently on Friday and Saturday. On Friday, a user is browsing through products while sitting in the office. On Saturday, a user makes a purchase in a relaxed environment,  while lying on the couch. Or vice versa. Actually, it doesn’t matter.

Visitors’ behavior differs throughout the week. It’s a fact proven by the following figures:



The table is taken from the report of a large e-store. On Thursday a conversion rate is almost twice as high as on Saturday.

If your test lasts less than a week, its results may be inaccurate. According to Peep Laja, an expert in improving a conversion rate and the founder of  ConversionXL agency,  an A/B test should last 2-4 weeks to deliver trustworthy results.

For example, Ton Wesseling of Testing Agency suggest to wait for at least 1 purchase cycle:

«You want to test as long as possible – at least 1 purchase cycle – the more data, the higher the Statistical Power of your test! More traffic means you have a higher chance of recognizing your winner on the significance level your testing on!»

6. How Much Traffic And How Many Conversions Are Needed To Perform An A/B Test Correctly?

Lack of traffic and conversions is one of the main problems of A/B testing on small websites. Experiments with small samples deliver results, which are highly likely to contain an error.

It’s often said about 100 conversions per original page and test page each. This may be insufficient, though.

If original and test pages have 100 and 110 conversions, accordingly, it’s difficult to draw a reliable conclusion.

On the other hand, if conversions differ drastically (e.g, 100 and 170 conversions) and the required conditions are met (an experiment lasts at least 7 days and statistical significance is over 95 %), making an accurate conclusion is rather easy.

Let’s take a closer look at statistical significance. A result is regarded as statistically significant, if there is a low probability of its accidental occurrence.  Each A/B testing service has built-in algorithms that automatically calculate significance. If you wish to double-check the result, you can use a calculator developed by KISSmetrics or by Evan Miller .

As you can see, you won’t be able to conduct an experiment with reliable results on a website with 1-2 conversions per day.

If you want to read more about rules of proper A/B testing, jump to the article by ConversionXL – Sample Pollution: The A/B Testing Problem You Don’t Know You Have.

7. Does A/B Testing Require Special Knowledge In Coding?

If you create tests by using Google Analytics Content Experiment, self-written developments, or self-written scripts, you will definitely need to know coding.

When using commercial services, you can  do  it without any coding skills in most cases. Unless you create really complicated experiments that involve JS/jQuery. For most experiments, a visual editor will suffice. All changes are made in WYSIWYG (What You See Is What You Get) mode.

8. How to Launch An Experiment Via Special Services?



After you’ve created a test version of your page in a visual editor, set a goal of your experiment,  chosen targeting and other additional options, you are given a special js-code. The code is added to the pages, for which you’re going to launch your A/B experiment. The code must be added in front of a closing tag </head>, otherwise your experiment won’t function or will function incorrectly.

Another important issue concerning code setting: do not set it via Google Tag Manager. If you insert a code via GTM, your page will flash while downloading, because the original page will be downloaded ahead of the test one.

If you have access to your website’s admin panel and know how to set a code, you will cope with that task in a minute. Otherwise, you’ll have to ask a programmer who is responsible for your website. The good news is that the codes provided by commercial services are set only once. Thereafter the experiment is launched automatically.

After the code has been added to a page, the traffic is distributed 50/50 (if you’re testing 2 variants of a page – an original one and a test one). Traffic is distributed via service algorithms automatically and totally randomly.

At  the early stage of an experiment, the number of displays of test and original versions may differ. But as the sample is getting larger, those figures are becoming practically equal. It’s the same as with tossing a coin.  If you toss a coin many times, the number of heads and tails will be practically equal. If you toss a coin 10 times, the same result may come as many as 7 times.

Important: Before A/B testing you should predetermine a sample size for your test and a point at which your test will end.

9. Do A/B Tests Affect Website’s SEO Negatively?

A/B tests have no influence on a website’s SEO whatsoever.

That fact was proven in  2012 by a Google employee.

A test page is downloaded dynamically, when a visitor lands on a website. It is displayed via Javascript and cookies. Googlebot and other search engines, that do not support Javascript and cookies, see an original page only. This is why you shouldn’t worry about that when making your A/B tests.

10. Does A/B Testing Slow Down Downloading Speed of a Website?

When using Google Analytics Experiment for A/B tests or any other A/B testing services, a website is downloaded with regular speed.  This is ensured by an asynchronous code used by all A/B testing services.  The code does not slow down downloading of other elements of a web page.

11. Where Do I Take Ideas For A/B Testing From?


You can borrow ideas for your A/B tests from web analytics, analysis of user behavior and interaction with users, surveys, heatmaps.

This would be a perfect scenario.

For really good ideas, you should use extra services, such as Google Analytics, Hotjar, Crazyegg, Yandex Metrika , Qualaroo, Usabilitytools, Survicate, Usertesting etc.

A typical example of idea generation for A/B testing:

You have a registration form on your website. Go to Yandex.Metrika and use Form Analytics option. Analysis shows you that most people experience difficulties with filling out the field “Company.” 50 % close the website. To fill out that field, the other 50% spend twice as much time as to fill in other fields.

What conclusion can we draw?

Upon facing that question, your potential clients find themselves at a loss. Unless the information is of key importance, you can remove that field from your test page and see how it affects the number of registrations.

That was a simple example of a A/B testing hypothesis based on your website analysis.

Another option is to use ideas that work for other websites – list of ideas and cases for A/B tests. But it’s not a good way for creating A/B testing with significant results. 

I wouldn’t recommend you to fully rely on such lists of ideas, though. While they can channel your own ideas into the right direction, borrowing such an idea and applying it to your website is  not the best thing to do.

You should analyze your website and your visitors’ behavior and communicate with both potential and existing clients. Thus you will know exactly what is wrong with your website and what can be done to improve the situation.

12. Can I Rely On Results Of  A/B Experiments Made By Other People?

Source: http://www.nailedit.ch/

You can and should analyze other people’s cases. Especially if those are the cases of your competitors or companies that work in an adjacent field or share the same target audience with you.  However, relying fully on other companies’ cases and ideas is a bad practice.

Something that works for one website is not necessarily good for another. Each website is individual. Each website has a different audience and interacts with it differently.  Implementation of a someone else’s successful case may even have a negative effect on your website performance. This is why you should never fully trust the results of someone else’s tests.

Make your own tests and use them to draw conclusions about effectiveness of your changes.

13. What A/B Experiments Are The Most Successful?

A detailed answer to this question was given by A/B testing experts in this article.

To put it in a nutshell, the following ideas turn out to be highly effective in most A/B cases:

  • Static image with one USP is better than an automated slider with several USPs.
  • Menu a la “Hamburger” + word “Menu”  is better than just “Hamburger”


  • Fixed header and footer with a CTA button (Subscribe, Order a Call, etc.) always improve a conversion rate
  • Having fewer fields in a registration form is better
  • A video clip describing a product/service increases a conversion rate
  • Contact information must catch a visitor’s eye
  • Live chat
  • Having real life reviews is better that a complete lack thereof
  • Possibility to make a purchase without registration. A visitor is invited to register only after buying a product as a guest.
  • Free delivery

14. How To Analyze Results Of A/B Tests?

When creating an experiment, you should always choose one goal (some services provide an opportunity to choose several goals at once) that will measure successfulness of your experiment. For example, clickability of the button  “See Price” (1st stage of the sales funnel).

That indicator plays a key role in analyzing your experiment. If it has improved and all A/B testing conditions are met – congratulations! You must have made a successful A/B test. But analysis is not finished, though.

Using Google Analytics, Changeagain.me or other services, where you can set GA integration, gives you an opportunity to perform  a detailed analysis of your experiment.

You can:

  • See how conversion rates of other goals changed;
  • Apply any segment to results of your experiment and perform analysis by individual groups of users;
  • See how key indicators of interaction with website changed (e.g,. bounce rate, average session duration, average number of viewed pages).

Analyzing the influence on other goals in the sales funnel is a very important issue when working with results of an A/B experiment. If it’s ignored, you can find yourself in a situation, where clickability improved at the first stage of the sales funnel and a conversion rate reduced at the last stage thereof.

Here is a highly demonstrative case to this issue, Most Crucial Mistake in A/B Testing.

15. Can I Make Several Tests Simultaneously?

Yes, you can. Note a very important issue, though. The audiences of your A/B tests must not overlap. If this condition is met, you can make as many simultaneous tests as you wish.

If the audiences do overlap, you have a problem. For example, a visitor sees a test version of a homepage. In a second experiment, a visitor sees an original version, and in a third experiment – a test version again. This is a rather confusing situation that will definitely affect accuracy of the results.

If you want to dive deeper in this issue, check out this article by ConversionXL

16. Can A/B Testing Deliver Quick Results?

No. One experiment won’t allow you to increase your website’s conversion rate or  improve your business performance.

A/B testing is not a magic pill that can solve all problems with your website. Most of your A/B tests won’t be successful.

To make A/B testing a success, you need to perform it regularly. You need to test a number of hypotheses, make conclusions, and continue testing. Only by doing this you can make A/B testing bear fruit.

Note that unsuccessful A/B tests is a great source of information about your target audience.  If your hypothesis has proven invalid, then the change does not affect visitors’ behavior. In this case you can strike off the hypothesis of your plan and make further steps towards understanding your audience.

17. How Often Do I Need to Make A/B Tests?

If would be perfect if you made A/B tests on a regular basis, one after another. Working process on improving your website’s conversion rate should look something like that: Making hypotheses – Classifying hypotheses by priority – Testing– Data collecting – Data analyzing – Testing of another hypothesis (based on results of previous experiments).

If you’re not making tests on your website, you’re missing daily opportunities to increase your website’s conversion rate.  Web traffic is coming, but you’re not using it to your advantage. It’s something like missed profit: your business is prospering, but it could be better.

18. How Is A/B Testing Different From Multivariate Or Split Testing?

A/B testing: In your test variant (variants), you change only one element (e.g, the button “Place An Order” instead of  “Buy”)

Split testing:  Your test variant (variation) contains a number of changed elements and is usually very different from the original variant. In split testing, changes are not made in a visual editor, but are prepared by a website’s developers.  A test page is assigned with a link a la Changeagain.me/variation1 and traffic is distributed between the original page and variation (variations).

Multivariate testing: Your variations include different sets of changes. For example, 4 variants of a headline message and 6 variants of a button make 24 test variants). Here is an example of a multivariate testing of Barack Obama’s presidential campaign.

Source: Optimizely.com
Source: Optimizely.com

Multivariate testing has a drawback, though.  It needs a large volume of traffic to be implemented. A very large volume of traffic, if you have 24 test variants, like Obama has.

19. Besides Pages, What Else Can I Test?

A/B tests can be performed not only on websites. Here is what you can also test:

If you have more questions on A/B testing, feel free to ask them in the comments below! We will add answers into the article right away!

Do you want your colleagues and friends to know more about A/B testing and improving a conversion rate? Just share this article with them. They will be grateful.