Email A/B Testing: The Basics

When it comes to email marketing, it’s easy to get overwhelmed. That’s why I like creating posts on the basics from time to time.

In this blog post, I’m going to give you the rundown on one of the most valuable marketing tools for your marketing campaigns: A/B testing. Other names for this winning metric method include “B testing” and “split testing”.

This technique is a targeted approach to improving click-through rates and email performance. With A/B testing, marketers are able to determine how to boost engagement, create more conversions, and generate more traffic for a website.

Enough stalling, though! Get ready to absorb! Let the games begin, and may the best version win. (You’ll fully appreciate that little joke by the end of the post.)

‍

The ABCs: Exactly What Is Email A/B Testing?

I’ll get this blog post rolling with the question you’re probably here for…

Exactly what is A/B testing? What does it do? How does it work? What does the data collected by these tests do for my email campaigns?

All of these questions are essential to understanding what A/B testing is, but we’re getting a little ahead of ourselves! Let’s take it slower, and I’ll start with…

What Is A/B Testing?

A/B testing is a way to compare two different versions of the same email to see which version yields a better response. It's a great tool to increase your open rates, click-through rate, and —ultimately — your conversion rate.

Why Should Email Marketers Use A/B Testing?

Think of an A/B test as a preparatory campaign report.

With one of these performance tests, businesses are able to get a preview of their future email performance metrics. The data collected during this process can then be used to tweak larger campaigns.

Depending on the goals of the email campaign, a well-vetted and thorough test group can help marketers create a call to action and streamline the conversion process.

‍

How Does A/B Testing Work?

Now that we know what A/B testing is, it’s time to learn how it works.

When marketers are brainstorming their campaigns, they will frequently create different versions of a message.

The ‘A’: Your Control Version

As its name implies, A/B testing relies upon having two versions of a message.

The first half of this equation will be your ‘A’ message, which is known as the control. This is what your marketing team is already sending out. As an active campaign, it will generally have the largest sample size, and it will serve as the baseline for each future test version.

The B Test: Your Experimental Version

Now that there’s a baseline, it’s time to create the test. This will be the second version of your email. In the B test, different elements of the email will be switched up.

The most basic form of A/B testing will only have one variable or, in other words, variation. It may not be a significant change, but even the tiniest difference can have statistically significant results! Some examples of elements that might be changed include:

• Call-to-action text
• Links for landing pages
• Tone
• Word order

Multivariate Testing

In a more complex version of email A/B testing, marketers will change many different elements in an email campaign. This may be done by using two versions of the email, — presumably, Version A and Version B — but it can also be done by making the control a separate email altogether.

That sounds a bit confusing, so let me clarify…

Let’s say that you want to change your email’s subject line and design. You already have a fairly successful campaign running, and you want to build off of it. In this situation, you’ll keep your current campaign as the control, aka Version A.

Now, you change the subject line. This will become your Version A email. Your Version B will have a different email design.

Because your control email campaign has already been sent, there’s no need for another ‘A’ test. Instead, you can now compare results from three different iterations of the same email by utilizing both your A and B test.

Multivariate testing is a lot more complicated than simple single-element tests. Successfully running email campaigns that are built off of the results of a multivariate test is tough, and it takes a lot of experience to truly understand the data that you collect.

When people are in the earliest stages of their email marketing journey, I always advise them to stick with basic tests. Begin with small steps and test one variable at a time. As you gain more experience, you’ll better understand the data you collect.

Moreover, you need a good email service provider to run proper multivariate testing. Without the right service provider, you won’t be gathering accurate results. This means that all of your hard work will ultimately be based on gut feelings, and that’s not very useful for anyone!

(Now, there are a lot of different email service providers out there, and I can’t possibly cover them all in this post. If, however, you were to ask me which provider I recommend, I’d definitely have to say Klaviyo! At The Email Marketers, we utilize Klaviyo, and its comprehensive data analysis tools are some of the most influential forces behind our most successful email campaigns.)

‍

How Do I Use Split Testing in My Email Marketing Campaigns?

The beauty of email testing is that it can be used to test many different elements of your email campaigns. From subject lines to email design, and every open rate booster you can think of, you can optimize your strategy by further and further refining your emails, resulting in a rock-solid campaign.

Simply put, “split testing” is just another way of saying “multivariate testing”.

Because of the vastness of A/B testing and marketing campaigns in general, I can’t actually tell you how to run your email campaigns. I can’t tell you what you should do, nor can I tell you which strategies will yield definitive results from the most subscribers. I can, however, explain some of the most common test elements that A/B testing addresses!

The “From” Line

When you receive an email, you want to know who it’s from, which means you’ll be checking the sender’s information. In fact, on many devices, the sender is the first tidbit of information that a recipient sees.

With that said, it’s worth remembering that simply putting your company name isn’t the only way to convey this information. Test different variations! There are many ways for brands to display this info, and knowing which approach is right for you will require a bit of company soul-searching.

To wrap this all up, I’ve gathered some of the most common approaches that companies have for their “from” lines. In these examples, we’ll assume that the brand is called “Gummy Candy”…

• Gummy Candy (the easiest approach!)
• The Gummy Candy Team
• Customer Service at Gummy Candy (Department + Company Name)
• Melanie at Gummy Candy (Staff Name + Company Name)

Email Subject Lines

As you begin to develop your email marketing strategy, you’ll want to pay attention to your subject lines. Think of these as your attention-grabbers. A good subject line is worth weeks of intense copywriting, and truly amazing subject lines can easily make your open rate skyrocket!

During the creation process, marketers will often weigh a variety of subject lines against one another. Even the smallest change in an email campaign’s subject line can yield much a better response from customers.

Tip From a Pro: Recent updates to different devices will impact your data collection. Apple’s iOS 15 update, for example, has made open rate statistics a fairly iffy metric.

Preview Text

Like subject lines, the preview text is one of the first glimpses your subscribers have of your email, which means that even the slightest change can dramatically alter your open rates and click-through metrics. Moreover, — alongside the “from” line and subject line — it is one of the three first impressions that customers have of your email campaign.

Tip From a Pro: The availability of preview text varies between devices and email providers. Most providers have added preview text as a feature, but some older software still withholds that information.

The Call-to-Action

Before you get too attached to your email design and content, you’ll want to run tests on your call-to-action. This is the handy button that users click to reach the relevant landing pages.

If you want to optimize your click-through rate, then you’ll need to run your CTA test early! Consider a variety of variables when you’re tweaking your campaign’s CTA, such as:

• Appearance (button style, color, and so on)
• Email design
• Placement
• Size relative to the rest of the content
• Wording (for example, “Buy Now” versus “Shop Now” versus “Check This Out”)

Campaign Personalization

One of the handiest tools that email marketers have is personalization. The overall success of an email marketing strategy will often hinge on the successful application of campaign personalization.

Now, here’s a big myth that I frequently find myself busting…

Campaign personalization doesn’t end when you add the customer’s name.

On the most basic possible level, you’ll want to know where to put the personalization. Generally, the name will be placed in the email subject line. However, depending on the campaign’s goal, the winning version may just have the personalization elsewhere.

With a bit of skillful segmentation, email campaigns can be further divided into different groups. Instead of sending the same email to your entire list, you can create different iterations of your campaigns. Each of these versions can then be targeted toward a different market segment based on factors such as landing page behavior, lifetime value, and past purchase history.

For the best results, you’ll want to have a decent sample size. Keep tabs on each campaign’s click-throughs and conversion rate.

Copywriting

This is where marketers dig into the nitty-gritty aspects of email marketing campaigns. You may have an absolutely stellar subject line, but it needs to be paired with a fantastic email body.

Testing an email campaign’s copy generally boils down to fine detail work, such as word order.

Headers and Subheaders

Utilizing headers and subheaders isn’t limited to long-form email campaigns. Once you’re satisfied with your open rate, you’ve reached the other half. You’ve made a huge step towards finding a winning email. You’ve engaged with your customers. Now, your job is to provide the most coherent and legible email content.

Having passed the subject line and entered the actual email, the headers (and, depending on the depth of your marketing campaign, your subheaders) are what draw consumers into your content. Proper headings and subheadings are just as important as subject lines, as they will be the main thing drawing customers to your call-to-action.

Tip From a Pro: While headers are essential to boosting your overall click-through rate, keep in mind that they are also an accessibility tool. If you want your transactional emails to reach more potential customers, take some time to learn about proper header formatting.

‍

What Other Elements Can I Test?

Now, this may seem like the end of your A/B testing. Your test group has been formed, and you’ve written up some amazing email campaigns. Your landing pages are ready, and you’re eager to push these campaigns out.

Wait!

There are still a few more factors that you should consider testing, and these seemingly small tweaks have the potential to yield more traffic and better results.

Length

Once you’ve optimized your open rates, you need to refine your conversion potential.

One of the most effective ways to do this is to test your audience’s tolerance for length. Some subscribers prefer long emails, and others want bite-sized content.

Tip From a Pro: Always keep in mind that different devices will display emails differently. The longer your campaign is, the more likely it is to get distorted when viewed on a mobile device. Take this into account when refining your email design on long-form copy content.

Plain Text or HTML?

It may seem like you should always be using the latest and greatest HTML tricks. Many people assume that an email performs better when it’s packed with flashy images and animations. However, test after test has demonstrated that this isn’t necessarily true. In fact, in some cases, a campaign’s open rate may suffer when too much HTML is used.

Before you make your decision, run an A/B test. Pay attention to the results. Some customers will prefer a plain text version over an elaborate HTML creation.

‍

How to Run A/B Tests

Now that I’ve explained what A/B testing is, it’s time for a brief overview of how you optimize your email campaigns using this approach!

1. Create Your Hypothesis

The first step of A/B testing is the hypothesis. Think of this as the reason you’re testing your messages.

This is your theory. I obviously can’t tell you what your theory is, but I can provide an example or two. When starting an A/B test, marketers may devise a hypothesis such as:

• A different call-to-action will improve a campaign’s click-through rate
• Personalization in the body of the message will be better than using the subject line
• A plain text version will perform better than an HTML email
• The word order is negatively impacting the conversion rate

Keep in mind that you don’t need to have an outcome in mind when you’re performing an A/B test. In fact, you may simply want to test whether or not one variable can have a statistically significant impact on your overall campaign’s performance.

2. Determine Your Objective

Once you have a hypothesis, it’s time to determine your objective. In most cases, this will be part of your hypothesis. For example, if your theory is that your CTA is too long, then your objective is garnering more click-throughs.

Your objective will be your experiment’s Version B. In other words, let’s use the same example — the overwrought CTA. In this instance, your B test will have a shorter CTA than your control.

3. Decide On Your Variable

Once you have your objective, it’s time to determine what you’ll be changing. In other words, you’re going to be finding out how to get into your reader’s brain. In order to do this, you’ll want to determine which variable is the most effective way to achieve your goal’s outcome.

That’s a little complex, so let me give an example or two of variables…

• To increase your click-throughs, you might change the CTA or headers
• Improving your conversion rate may mean altering the email design
• Engagement might be improved by changing the preview text
• To boost the open rate, you might change the subject line

4. Decide Where to Split

The last step of the A/B testing process is splitting.

This is where you’ll determine your sample size. To split your list, you’ll need to look at your email clients and determine how you’ll divide your subscribers.

The simplest solution to this problem is to split your entire list 50/50. However, the options are limitless. Divide your audience as you see fit!

For example, you might send the control email to 25% of your subscribers, make the test group another 25%, and from there send the winning version to the remaining recipients.

There are multiple ways to get your sample size, and you might even have a control, a version A, and a version B in play all at once. No one way is necessarily preferable, as long your company is equipped to tackle the results.

5. Run the Test

Once you have everything figured out, it’s time to run the campaign. This is a love it or hate it thing. Some people will love doing this, and others will find it tedious. If you really feel out of your depth on this, consider outsourcing this part of your marketing so you're confident that you're testing the right variables to achieve your goals.

For the most accurate and actionable data, you’ll want to run the test for at least two weeks.

6. Analyze the Results

When it comes to analyzing results, statistical significance is the name of the game.

This is the arduous, tedious part of the process. You’ll need to really get in there and scrape at the data. Examine all of your relevant information — the open rates, subject line engagement rates, conversions, and so on — and draw conclusions from it.

Understanding this part of the process is hard. Not everyone can do it, and that’s why I always recommend that companies seek out professional help.

7. Keep Going! Rinse and Repeat

Once you have a winner from your first A/B test, it's time to go back to the drawing board.

You can either choose another variable to help get you closer to your original objective or create another objective and develop a new test from there. You should be running A/B tests consistently as part of your general email hygiene.

‍

A Few More Considerations for A/B Testing Your Email Campaigns

As you run your A/B test, you’ll also want to keep a few things in mind.

These variables may sound inconsequential, but even the smallest errors can dramatically impact the accuracy of your A/B test!

Size and Timing Matter

Much like any study, you need a good sample size to get good results. In the world of A/B testing, the size of your test group matters. Larger groups yield more accurate data. Moreover, you want to make sure that your test group is entirely random. If you want to get statistically significant results, then your empirical data must be gathered from randomized groups.

Conflicting Data

You’ll also want to take confounding variables into account as you prepare to run your test.

Confounding variables are factors that are outside of your control. A very relevant example of a confounding variable is the pandemic, which dramatically weakened consumer morale. More commonplace examples include timing, individual customer reception, and recent events.

‍

Tools For Testing

There are two ways to run these tests: old and new school.

Both of these methods have their advantages and disadvantages, and neither is necessarily the “wrong” way to run a test.

New School: Automation Software

These days, most email marketing software comes with its own automated tools for running A/B tests. Taking this approach is simple, easy, and affordable. With automated testing, users are able to quickly and easily run a test.

Old School: Manual Testing

Some people prefer to manually run their tests. This approach is slower and harder, but it offers an unprecedented amount of control over your test. You get to decide how to split your test group and send your emails. You get to handle the results. If you’re more interested in data or don’t have easy access to email marketing software, then this is your best option.

‍

The Best Practices for an A/B Test

This is, quite honestly, a lot of information to take in. There are plenty of moving parts involved in running a proper A/B test, and understanding all of them is hard. To help make your email campaigns really thrive, however, I’ve rounded up some quick tips on running an A/B test.

• Larger Test Groups Are Good: The larger your sample size, the better your email marketing campaigns will be. You will have more empirical data to work with, which means your results will be more accurate.
• Run Tests Simultaneously: Never space out your tests. Results will only be accurate if emails are sent out at the same time.
• Test Early and Frequently: Never rely on one test, and always run tests early.
• Trust Your Data: Never trust your gut feelings in marketing. Your job is not to guess! You want to base all of your decisions on solid data, which means that you’ll want to invest time in studying your metrics. What are your click-through rates? How often are people visiting our landing pages? All of these factors are huge parts of A/B tests.

‍

Why You Should Be Running A/B Tests

After all of this information, what does it all mean for you? How can an A/B test help your company thrive, and what does this marketing strategy have in store for your company?

At the end of the day, knowing things doesn’t matter as much as getting results. To wrap all of this up, I’ll highlight a few basic benefits that an A/B test has the potential to yield, such as…

Better Conversion Rates

No matter what you’re doing or selling, your bottom line is the conversion rate. Everyone wants to increase their conversion rate. Fortunately, an A/B test is the perfect way to find out how you can achieve this goal. A solid test strategy gives you the data you need to tweak your email marketing campaigns to perfection.

Know Your Target Audience

Email testing helps you understand your target audience better. You’ll get a feel — better yet, you’ll get the DATA — on your subscribers’ preferences. You work hard to create great emails for your contacts, so why not tailor your messaging accordingly based on data?

Stay On-Trend

As with clothing and music, marketing trends go in and out of fashion. (Of course, there are some "classics" in marketing, like a fantastic subject line! Like your morning coffee, you will always need a good subject line.) Use your B test to see if that new trend you saw or read about resonates better with your target audience.

‍

Amplify Your Success With The Email Marketers

When you’re ready to jump into the world of A/B testing, things can get tricky. You’ll be juggling countless data points — more click-throughs, decreased open rates, more traffic to our landing pages — and it can get wildly overwhelming.

Sometimes, you need to enlist some outside help.

As you develop your marketing strategy, remember me and The Email Marketers. We’re always here to help you, and we love seeing small businesses thrive! We’re here to help, and I love helping people. If you’re looking for even more guidance on digital marketing, — from how to run an A/B test to finding the perfect digital marketing agency — I’m always here; so, don’t be afraid to drop by my blog or send me a message!