A/B testing hypotheses don’t come easily. Sometimes it’s hard to produce them because of lack information, data and creativity.
First, you need to realize why you want to run an A/B test. More conversions? Sign-ups? Better user experience? Then, when you establish your motivation, it only gets worse.
You’ll listen to advice that red wins over green and you’ll start changing all your CTA’s. It will or will not work. Then you’ll read that forms on the right convert better, and you’ll start another test. If something worked for them, it doesn’t mean it will work for you.
What experts say?
We asked industry CRO experts about their experience and best practises.
“In our company we leverage a lot of data such as analytics, analysis, as well as heat/click maps. We typically find bottlenecks or friction points based on this information.
What people are paying attention to most as well as what they are ignoring. From this we back it up with our best practices based on thousands of tests and hundreds of campaigns.”
“I find there is never a shortage of ideas. The problem is more around prioritization. Which test will yield the biggest return?
That opens up a big can of worms in making sure you and your team knows what they can effect and with what magnitude. Things that often come only from experience.”
“The best way to generate hypotheses is from customer research. Do customer development calls, surveys, or any other qualitative method to get into the heads of your users at each step of the funnel.
That’s how you come up with ideas that turn into major wins. Whenever I get stuck, I always find a way to talk to my users. Then my win rates and conversion lifts always see a big jump.”
How we used HotJar for analytics and generating hypothesis?
It’s true. To know what happened on our own wesite we tried to utilize HotJar tools. It’s easy-to-use and easy-to-integrate analytics tool that includes heatmap, visitor playback, conversion funnel and surveys.
Heatmaps of your key pages
Your high traffic and high bounce rate pages offer you the biggest opportunity to retain more visitors on your site. Setting up heatmaps for these pages allows you to identify how your visitors are engaging with your site pages – and the biggest opportunities to keep more visitors on your site.
Start off by creating heatmaps for your high traffic and high bounce rate pages. These offer you the biggest opportunity to retain more visitors on your site.
There are a few heatmap test provided by HotJar.
- The Link Test – Are visitors clicking on dead ends?
- The Distraction Test – Are irrelevant elements distracting visitors?
- The Information Test – Is there missing information from your page?
- The Device Test – Does your page work on different screen sizes?
- The Depth Test – Is all your content easily reachable?
- The Engagement Test – What are your site visitors really looking for?
- The Fold Test – Does your page have what it takes to keep your visitors?
- The Header Test – Is your page header helping or hurting your site?
Generating a heatmap for your high drop off points in your funnels and other high exit pages is very insightful. It will give you a better idea of what your visitors are actually seeing and what they are ignoring just before they abandon your site – inspiring tests and changes to your pages.
Use a Web Analytics solutions such as Google Analytics to identify key pages. Use ‘Behavior’ > ‘Site Content’ > ‘All Pages’ and identify pages with high Bounce rate and % Exit rate.
Expert’s note
“I do regularly use heatmaps and scrollmaps for generating hypotheses. It’s a great way to see if people are clicking what you think they’re clicking on. You can then use that data to brainstorm ideas for new tests.”
Uncover user behavior
Visitor Playback is all about putting yourself into your Visitors’ shoes. By replaying recordings of your abandoning Visitors you can start to see how your Visitors react to your pages. Are they getting stuck at a particular step? Do they abandon the page very quickly?
Tips for reviewing recordings
- Filter out the right recordings by entering your main ‘barrier’ page url into the ‘EXIT PAGE’ field in the filter. (barrier page = where visitors are bouncing)
- Filter by device and compare the different behavior. This will allow you to identify opportunities to improve your site design across different screen sizes.
- Tag recordings with names for issues or concerns so you can easily find them later.
- Replay sessions of visitors that converted by adding your thankyou / success page in the ‘VISITED PAGE’ field in the filter. Take a note of their most common steps and the content the visitor engages with. Then improve your pages by making it easier and emphasizing these steps and content.
Expert’s note
“Session recording is also valuable. But I prefer to do 1-on-1 screen shares with people that I know are in my target market. The ability to ask followup questions is immensely helpful.
Every company I’ve ever worked with would see huge wins from doubling the number of customers that they talk to individually on a weekly basis (and most companies don’t talk to any).”
How to discover your visitors?
By interviewing your largest pool of visitors you will get a quick understanding of why your visitors are on your site. Make sure to dig deep why your visitors are using your site. If you are running a gaming site you want to ask why they want to play – is it for money, fun, to socialize?
This can be used to make your site’s design and text more relevant and persuasive.
By asking direct questions to your visitors you will get a quick understanding of why they are on your site… and what can make them stay longer. The answers you will collect can be used to make your site’s design and copy more relevant and persuasive.
By asking direct questions to your visitors you will get a quick understanding of why they are on your site… and what can make them stay longer. The answers you will collect can be used to make your site’s design and copy more relevant and persuasive.
So create a Feedback Poll now asking one of the following question (‘Long text answer’):
- “What led you to look for [service / product type]? Explain in as much detail as possible how it can make a difference to you.”
- “What nearly stopped you from using us? List as many items as you can think of.”
- “What persuaded you to [action e.g. buy / signup]. List as many items you can think of.”
- “What could we have done to make your decision easier?”
Important tips…
- Use an open ended ‘Long text answer’. This will allow you to get juicy answers rather than forcing your visitors to choose a pre-existing answer you created.
- Trigger the poll to appear after 5-10 seconds to maximize replies.
- Read through your responses and make sure that on your site you are using the same words / terminology your visitors use.
- Read through your responses and make changes to improve the site experience or include counter arguments to your visitors’ objections. Remember to be bold!
- Give an incentive by offering a gift or a prize to one lucky winner.
- Give a short deadline to receive responses to add urgency.
- If you don’t have many existing customers / users then setup your survey to appear on your ‘thank you’ page.
How to find the biggest barriers on your website?
If you don’t understand where and why your visitors are leaving your site you cannot really improve your site’s experience and bottom line. So don’t make the mistake of working on too many parts of your site at one go.
By creating Funnels you can understand where you are losing the most visitors allowing you to focus on those steps first.
The key to a great Funnel is to build it ‘backwards’. Start by asking yourself – what is your biggest goal? A signup, an order? Create a Funnel that maps back to your highest traffic pages.
Important tips…
- Each Funnel will only show data for Visitors that visited the first step in your Funnel. So if you have 2 main sources of traffic for a goal e.g. Homepage and Landing page, you will want a separate Funnel for each.
- Funnels in Hotjar are not based on reports – they present data retroactively based on a sample of your traffic.
Recruit Users for quick Usability Tests
User testing is the process of looking at your users while they use your site and comment on the experience as they go along. Nothing quite beats watching and listening to your visitors while they use your site.
Although this might sound like a lot of work, it’s actually quite a fun and extremely insightful experience.
How to run user tests:
- Start by setting up a recruit form on your high traffic landing pages. Give Visitors an incentive to sign up for testing e.g. free content, an amazon voucher or early access to a new feature.
- Once you have users signing up for users tests you need to choose some users to reach out to. You want to aim for 5 user tests so you should reach out to 10-15 users from your list.
- Contact your final list via email or phone and propose a date and time to screen share. You will want to use Skype or a tool like GoToMeeting so that you can see your users’ screen.
Important tips…
- Move fast – the quicker you invite your visitors after they have signed up for testing, the higher the response rate.
- Prepare a list of steps / goals you want them to complete, as well as a series of questions that will allow you to uncover Barriers.
Case from Bartosz Mozyrko, Usability Tools
Each website is a unique environment influenced by factors native to it. There are no universal truths.
But there’s data. Hard, tangible data about your users that you can use in improving your pages. They are right in your unique environment. At UsabilityTools, we know that studying recorded user actions is the best source of hypotheses for tests.
A gamification website GetBadges wondered why their conversions are so low. Instead of deciding on an overhaul of a website or product, they observed with our help what users do on their websites. To their surprise, they learned a single button was causing people to leave the register form. They clicked it hoping to finish the register process only to find their page reloaded. Frustrating.
GetBadges removed the button after testing and increased conversions by 40%.
Google Analytics would tell you that only a click happened, but would you make a connection between a click and an exit? Would an article on the Internet consider that there is a single button on your specific form that you should change? No.
But data coming from observing users would. So, for me, the golden rule in coming up with hypotheses is observe your users and find their pain-points.