The Ultimate Guide to A/B Split Testing with WordPress
Does the thought of your website keep you up at night? Do all the “what ifs” have you worrying about the design, the functionality, the calls to action, the wording and, ultimately, your conversions??
You could have your designer/developer make changes based on what you think might help with improving the number of site visitors, the time they spend on site, as well as their conversion rate. But what if you’re wrong and you spend all that time redesigning your site only to find that they actually liked the way it looked before?
You know that your website serves many purposes. One of those is to showcase your business and the solution or service it provides to customers. Another is to give your customers an easy way to reach you in the form of a phone call, email, form submission, or online purchase.
If you’re worried that your website may be standing in the way of successfully making this connection with your audience, then it’s time you took a look into the underlying cause and devise a solution.
What Is A/B Testing?
Supposing your website is causing you stress, you could leave it be and try to compensate from its lack of performance by investing in other ways to target new and existing customers. That wouldn’t make much sense though. Your website is the online face of your business. You need it to perform just as well as anyone else who works for your company.
You can’t assume that because you love your website and you think it looks great that your online visitors feel the same way. More importantly, a nice-looking website is only part of what’s needed here. It needs to be easy to navigate, have a logical flow, and clearly explain what you do and what you offer. No matter how excited you are about your website, if it’s not performing well or you want it to do even more, it’s time to make a change.
According to Merriam-Webster’s dictionary, the Scientific Method is defined as:
… Principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses.
A/B testing is exactly that. It’s a digital version of the Scientific Method whereby a website’s design, layout, copy, and/or functionality can be tested without permanently altering the makeup of the site. By establishing very specific variations on pages of your website, you can study the resulting differences in behavior of your visitors and consequently make well-informed decisions regarding what improvements are needed.
There are a number of approaches to consider when conducting A/B testing:
- Split Testing: This is the typical method when someone refers to “A/B testing.” A singular element on a page is chosen to be altered and then two versions of that page are created with an A variation and a B variation. Those two-page versions are then shared with different visitors in order to test the varying responses between the two.
- Multivariate Testing: For those who have successfully conducted A/B tests in the past and want to look into whether more complicated pairings of elements on a page may affect the user experience, that’s when multivariate testing comes into play. Rather than focus on changing one element, a variety of elements and combinations of those elements are reworked simultaneously.
- Experimental Testing: In this method, all elements on a page are under consideration for testing, so this is really only for users who have extensive experience conducting multivariate tests and understand what to do with the results.
Whether this is your first time with A/B testing or if you already have experience in this, split testing is the optimal choice. It ensures a level of simplicity and focus that allows you to work with a control (your original design) and a variation (your new design). By testing two very clear differences among your audience, you can identify changes in behavior (if any) as it relates to that one element.
It’s important to remember that while you may have a good understanding of your audience as a customer, you may not totally understand what they want or need from the online experience. Rather than try and make that decision on your own, A/B testing can help you gain those insights. With carefully measured steps and results, you don’t have to worry about whether you’ve made the right call or not—the data will let you know what to do.
When Should You Conduct A/B Testing?
Now that you know what A/B testing is, it’s important to consider the “when” of the thing.
Most people probably assume that A/B testing is meant solely for websites that are underperforming. While A/B testing may be one solution for that, it won’t always be the case. If oversized images are bogging down your site’s loading time or if you have broken forms and incorrect contact information keeping visitors from reaching you, there is no amount of A/B testing that is going to fix those problems. Remember that you need to employ the use of the right tools for the right problems.
When it comes to A/B testing, there are a number of scenarios where you may need it:
- Site re-launches: Rather than wait to see how a new site design (or messaging) performs with your audience in the long-run, start with split testing as soon as possible. There was a reason for your website’s re-launch, so you should do what you can this time around to better shape all the pieces of it based on what your visitors want.
- Marketing campaigns: Marketing campaigns can be a highly effective way of expanding your business’s reach. However, these can often take a lot of time and money to strategize and create. While in the design and copy phases of your campaign, create A/B tests for your landing pages. In the off chance one of your ideas doesn’t go over well with your audience, you can potentially save this marketing venture from being a total loss.
- Decreased conversions: If you’re noticing a change in the amount of website conversions, now is the time to take action. If the problem isn’t due to a back-end issue, A/B testing may help you sort out where things are going wrong on the site.
- Increased conversions: While increased conversions are definitely a cause for celebration, this is also a great time to work on keeping that momentum going. Conduct some A/B tests (one at a time) to figure out what is working and what your visitors would like to see more of across your website.
- Product launches: When releasing a new product or service, there is never a guarantee it’s going to perform as well as your current offerings. Much like the site re-launch, you don’t want to have a cross-your-fingers-and-see approach. You want to set yourself up for success and test out a number of options during the launch so you can act immediately if something about the page or information isn’t working.
Ultimately, the choice of which elements to test will fall to you. You’re looking to make changes to your website for a reason, so you must have a gut feeling that something isn’t working the way you need it to. Based on where the problem lies—a high bounce rate on a specific page, a lack of conversions on a certain form, confused messages from visitors who can’t find something—you can then make a determination of where the test should occur.
Once you’ve got the page figured out, you’ll need to determine which element you want to rework. It could be something like one of the following:
- Paragraph text
- Contact form
- Call to action text
- Call to action button
- Page length
- Social proof
- Awards and badges
And within those elements, it could be any number of issues causing disinterested visitors:
- Distracting color choice
- Too many steps required
- Confusing button placement
- Negative messaging
- Too many ads
You may want to step things up a notch with more advanced test
Overall, you really won’t know what could affect your user’s experience until you get started with the process.
How Do You Conduct an A/B Test?
An A/B test runs just like the Scientific Method: research, hypothesize, test, monitor. So if you haven’t taken high school chemistry (or it’s been a while) here is a recap of all the steps you’ll need in your testing procedure:
The first place to start is by asking yourself questions: Why do you want to A/B test? Have you identified a problem page or process on your website? Why do you think visitors aren’t responding well to that page or process? How do you think it should be updated?
While it’s important to ask these questions and to know why you want to do this, you also need to conduct some research to find out a bit more about what’s going on.
- Start by digging into your analytics and see if you can identify any troublesome exit pages where activity and interest seems to stop.
- Analyze the popular pathways visitors take on your website. If they’re not doing what you want them to, then you need to make changes to better guide them.
- Use heat maps to see where people’s eyes are drawn to on your website. It may not be where you think.
- Conduct interviews with your sales or customer service team members to see if they have any insights on what your audience is thinking.
- Conduct online surveys with your audience and encourage them to provide feedback. That will be the most straight-forward approach to finding out what needs to be changed.
#2: Establish the Goal
Once you’ve identified a problem area on your website (or an area that you think could perform even better than it already does), you need a goal. When your team updates this page, what is the end result you want to see? More conversions? Longer time on the page? More visits to a different page that shoots off from this one? Without a clear and achievable goal, you won’t be able to verify whether the test worked or not.
You’ve identified your goal, so now you need a hypothesis.
Which elements on this page could stand to be updated? What sort of change could you make to each of those elements? What result would you expect to see with each of these variations? Why? Jot down your expectations for analysis later.
#4: Choose One Variable
You’ve compiled a list of elements on your web page that could potentially affect audience reception or behavior, so it’s now up to you to choose one element to focus on for this test. Perhaps it’s the color of the call-to-action button that just doesn’t stand out enough. Perhaps the form is in a weird spot that people don’t think to scroll to. Perhaps the headline at the top of the page is turning people off. Use what you know to identify the variable and determine how you will alter it.
#5: Develop a Timeline
A/B tests cannot run indefinitely. The point of these is to test a variation, monitor the reaction, and take action.
Depending on the amount of traffic to your website, you may want to allow the test to run for longer so you have enough activity to measure. Regardless of how long you feel your test should run, start with a test duration calculator to get started. This will help you determine where that sweet spot will be. Ending your test too early may lead to a lack of valuable stats. On the other hand, ending a test too late could be detrimental for a number of reasons. For one, the search engines may look unfavorably upon that (see more on that below). For another, if the variable was not a good pick, it may be the reason there are no results to work with and should be ended sooner rather than later.
When conducting a test, it’s important to develop a hypothesis so you have a clear vision of what you think will happen. It’s equally important to closely monitor the real results against the hypothesis in case there are severe deviations that require you to take action.
#6: Set Up
The next step is to create the variation on your page and track it.
For A/B split testing, you’ll need to create a duplication of your web page within WordPress and then apply the desired change to the new one. Your testing software or tool will then push part of your web traffic to the variation on a separate URL. The purpose of this test is to find out what these changes mean to visitors. In order to effectively test the reception of your page’s change, the variation should only be sent to new visitors. You don’t know if the alteration to your site will be a welcome addition, so it’s safer to test this on new visitors who have no preconceived notions about your brand.
When adding new pages to your website or making any changes that affect the user experience, it’s important to consult with Google. A/B testing can introduce some potential issues for you with the search engines, so keep the following in mind before proceeding with the setup and launch of your first test:
- Cloaking: Google is very sensitive about websites attempting to deceive the search engines for better search results. Cloaking is one such example whereby a split test is created to show one version of a website or web page to the search engines and a different one to users (which is a big no-no).
- Indexing: Any time you set up a duplicate web page, it needs to be set up for proper indexing (or none at all) by the search engines. Since the purpose of A/B testing is not to create a permanent duplication of your site’s content, your variable page should not be marked with the noindex tag. Instead, you should use the rel=”canonical” link attribute to properly establish its purpose.
- Redirecting: Another way you can indicate to Google that this is only a temporary change is by setting up the proper redirect. A 301 redirect indicates a permanent redirect to the new, alternate URL. A 302 redirect indicates a temporary redirect to the new, alternate URL and, in so doing, lets search engines know they should continue to direct traffic to the original URL.
- Conclusion: Your test needs to come to an end and, according to Google, within a reasonable timeframe. If the search engines feel your test is running unnecessarily long, they may interpret that as a form of deception and penalize your site. So make sure to closely monitor your testing tool to ensure you’ve collected as much data as you need and then close out the split test completely.
There are many tools you can use to help you set up and track the performance of your A/B test (see below). These tools may not help you account for Google’s tips though, so make sure these are built into your setup process so as not to run counter to what the search engines want.
#7: Run the Test
Using your A/B split testing tool, kick off your test. As with any true scientific test, consistency is key:
- Your control and variation need to be tested simultaneously. If they are not tested under the same conditions (especially the time period), the results may not portray an accurate representation of how your change was received. The only difference between your control page and the altered one should be the actual variation.
- For any update you make to the inherent design of your website–the color of a call-to-action (CTA) button, the layout or length or a form, etc.–you should apply this variation throughout all pages of your website where the element occurs. Even though you may be testing a specific page, you still want visitors to have a consistent experience across your website.
Once you’ve kicked off your test, make sure to regularly monitor the results. A timeline is established, but that doesn’t mean you need to wait until the very end of it to see how the page variations are performing (especially if something isn’t working or stops working in the middle of the test).
Monitoring isn’t just about logging into whatever tracking tool you’ve signed up with. If you have a customer service or sales team in place who gave you feedback while doing your initial research, they should be able to tell you if they’ve noticed any changes since the testing took place as well.
#9: Take Action
Once you have solid data to work with, take time to analyze it. What have you learned from the A/B test? Which variation worked best? Can you tell why? Is this change something you can apply elsewhere on your website? Is there a different element on this page that may still be causing issues? Was the difference in performance between the variations worth making any changes at all?
The results of your A/B test may be surprising. No matter how extreme or unexpected they may be, the results of the test–if conducted properly–don’t lie. So if you’re feeling wary about implementing a change that doesn’t sit well with you personally but that was well-received by your audience, don’t let your own aesthetic preferences get in the way. Remember that if the A/B test result goes live and does not work in the long-run, you can always change it back or set up a new test to find a more successful variation.
Once you’re comfortable with the process of A/B split testing and have a clearly defined approach, plan on making this a regular part of your website maintenance. With each test you’ll learn more about your audience and be able to more easily make decisions related to website and business improvements.
With a clearly defined approach and carefully measured results in hand, you should be able to more easily make decisions related to website improvements.
What Should an A/B Test Look Like?
There are no hard and fast rules about which elements you need to update for greater traffic and better conversion rates. That determination will be yours to make based on the current state of your site, the feedback and insights you have into its performance, and your ultimate goal in A/B testing.
While the choice to test a specific variable will depend on what’s happening with your own website, that doesn’t mean you can’t look to others for inspiration on what has worked for them. Here are some well-documented cases of brands and people that have used A/B testing to success on their website:
- Gyminee: The first and simplest of the tests conducted on this website were to the home page. The original version of the home page included 25 above-the-fold calls-to-action. In the B version of the website, they cut that number to 5 CTAs. They ran the test twice and the simplified B version of the home page resulted in 19-20% more conversions.
- Adore Me: This clothing retailer uses A/B testing to ensure that the product images they use on their website perfectly are framed. So when it comes time to shoot the photos of the lingerie and the models that wear them, they make sure to include variations in the model, the color of lingerie, the placement of the model’s hands, etc. This way, they have a variety of images to test out for every single product on the website.
- Soocial: Soocial wanted to experiment with conversion rates when changing up the language of their home page’s CTA button. They conducted a number of tests to find out what would perform the best. The design and words they originally had on the site proved to be fairly effective, but when the words “It’s free!” were placed beside the button, the conversions increased by 28%.
- ComScore: The number of demo requests on ComScore’s software product page were underperforming. They believed part of the problem was the lack of emphasis on the social proof (the testimonials). So they tested a number of layouts and designs against the original. All of the variations outperformed the control, but there was one clear winner they identified when all the testing was done.
- Barack Obama: Over 20 months, the Obama digital team conducted over 500 A/B tests on their campaign website. One of the examples they provided was around the decision to improve the quality and reception of the headline on a donation page. Without sacrificing the brevity of the message, they changed the wording to better connect the action on the page to the donation and saw an increase in 21% conversions because of it.
- The Olympic Store: The checkout process for e-commerce websites can vary. This A/B test proved that a single-page checkout had 21.8% more conversions than a multi-step, two-page checkout process.
- Sony: Experiencing a lack of success with their ad campaign, Sony hypothesized that two competing CTAs in the ads left visitors feeling conflicted. They created two new designs of the ads–to promote each of the calls-to-action separately–to run against the original. With a clearer directive and a better targeting of the audience, one of the tests resulted in a 6% increase in click-through rate and a 21% increase in visitors going to the shopping cart. The other ad tested did not fare as well, and provided Sony with better insight into which promotion was more attractive to their audience.
- ABC Family: This case study is the perfect example of why user behavior statistics can help create more effective tests if you have an idea of what drives your audience in the first place. When the original page design showing a promotional image for one of their shows didn’t perform as well as they thought it would, they turned to a text-heavy menu of options instead. Because they knew site visitors were more apt to do a search for the show rather than click on that image, the results of this A/B test were a huge success and led to a 600% increase in engagement.
A/B Test Ideas
Once you’ve got the motivation to conduct your own A/B tests and have found some inspiration, it’s time to start focusing in on the specifics that can be done for your website.
Here are some test ideas to get you started:
- Add a video testimonial
- Include logos from former clients or current partners
- Use a different promotional offer
- Use a different promotional photo
- Update the call-to-action wording
- Update the call-to-action button color or size
- Change the text layout from multiple columns to single columns
- Alter the headline above a form
- Shorten the length or appearance of a form
- Reduce the number of steps to checkout
- Revise the navigation structure
- Change the font size
- Use strategic bolding or italics in the copy
- Rewrite copy to include less paragraphs and more bulleted or numbered lists
- Revise the content that appears above-the-fold
- Change pricing or pricing structures
- Change ad content or design
For your first A/B test, start out small and simple. Make sure your research and planning is guiding you toward the right hypotheses and testing variables before committing to more drastic or extensive updates. If you find that the suggestions or case studies above aren’t applicable to your current situation or you simply want more help, feel free to check out Hubspot or Wired for more information
Many of the A/B testing tools we’re about to mention below (like Virtual Website Optimizer and Optimizely) also serve as strong resources on the subject of A/B testing. So regardless of whether you use them for testing or not, the information they produce around the topic is still worth checking out if you’re serious about this testing business.
What Are Some Tools for A/B Testing?
You don’t have to set up your A/B tests on your own. There are a number of services you can sign up with (most for a fee) that will help you create, conduct, and manage your A/B tests.
When looking up information on A/B testing, chances are good that you’re going to see Optimizely in the list of top solutions or mentioned as a reference for more information. With customers like The New York Times, Disney, and Priceline under their belt (and case studies to back it up), Optimizely is the A/B testing tool to trust. And it also helps that their Starter plan is free.
Nelio has two especially strong traits to pay attention to. First, they specifically target WordPress websites, and second, they have custom-tailored solutions for different industry and website types. When software providers are able to speak to the specifics of your business and thoroughly understand the platform on which you work, there is usually a level of expertise there worth paying for.
Oracle’s Maxymiser testing tool is another one that drills down into industry specialty. They’ve also included mobile app A/B testing which you don’t see in many other tools, so this is another differentiator worth considering if you have a large amount of traffic visiting your website from mobile devices. There isn’t any pricing listed on their website which means it’s probably more expensive than the competition, so be prepared for that before reaching out for more info.
VWO is another one of the leading A/B testing tools currently available. They’ve worked with some of the top brands around the globe and have the results to prove it (like the 62% increase in inquiries for Hyundai). If your website has a lot of web traffic and you can afford the higher price tag on these services, this is the tool for you.
If you’re already a fan of Adobe’s tools, then this may be the next logical step (and even if you’re not familiar with their other tools, this still might be the right solution). Their home page alone offers many tools to help you make that decision: an interactive demo, a self assessment, and an ROI calculator are all right there to help you get started.
Unbounce’s client list includes the likes of Hootsuite, Thomson Reuters, and Zoho and they’ve got the testimonials and statistics to prove that the solution they’ve provided to those companies has worked. With a specialization in the creation of landing pages that convert, Unbounce is a good tool to consider for your website and newsletter integration needs. They even have optimized landing page templates to get you started.
If what you’re looking for is less of an all-encompassing A/B testing tool and more of a means to find out where to start, Usability Hub offers a number of tests to help you identify problem areas on your website. Start with the Five Second Test and found out what first impression your website is making with visitors. You can move on from there to try out their other tests and discover what more can be done to your web design and layout to improve functionality, performance, and user experience.
Note: If you’re willing to pay extra for a testing service – which may indeed be the right solution if you plan on doing regular tests – make sure to check in the WordPress Repository for their associated plugins before you sign up.
For those of you who don’t want to make that level of commitment just yet, but still want to be able to run tests without having to update the coding of your website or set everything up from scratch each time, look into the following plugins to get started:
Google previously had their own A/B testing software called Google Website Optimizer. Now defunct, that functionality has moved over to Google Analytics. While Google Analytics is always a great tool for tracking your website’s performance (and is probably where you started seeing problem areas in the first place), it’s much easier to monitor your A/B testing from the source. With this plugin, you can use your Google Content Experiments within WordPress.
This tool is an all-encompassing call-to-action solution for WordPress. If you need help building CTAs on your website, this will guide you through the steps of creating and placing them on your site. In addition, there is an A/B testing feature that displays the details and results of your test in a very clear and easy to understand format.
This plugin is very similar to (and can be managed in conjunction with) the WordPress Calls to Action plugin mentioned above. Creating landing pages for your website can be a tedious task, so having a tool like this is invaluable. It will guide you through setting up a well-planned out landing page and provide you with testing opportunities within the same interface so you don’t need to recreate the page with the alteration from-scratch.
This is a plugin that allows you to create alternate headlines and titles for any post or page on your website. Blog post or page titles may easily be overlooked since design is often the first thing people turn to when there’s something wrong. However, better headline writing has proven to result in higher conversions. With short attention spans, readers need something to draw them in—a promise that the content will be worth it. That’s why headlines and titles should never be forgotten in this process.
If you’re looking for variations on a larger scale, the A/B Theme Testing tool helps compare your website between different themes—similar looking or completely different in design. The testing tool also helps do most of the work for you so you don’t have to go building a separate website with the new theme. Additionally, all analytics are included within the tool.
A/B testing is often focused on improving the user experience on your website. While that is definitely true, the underlying fact is that you need to do what you can to increase your chances to recover missed opportunities. By identifying and fixing small elements that may be distracting your visitors from taking a desired action, you are actively moving your website and business into a better place.
Your website is an important piece of your business’s identity and is often the place people go to learn more about you before seeking out your services or products. It would be a shame to invest all that time and energy into building this piece of your brand for it to not work properly. Through A/B testing, you can make sure your website—whatever state it’s currently in—is well-received and functioning at the highest level possible.
Image credits: Optimizely.