Note: A/B testing is rolling out to a small group of pilot accounts first. If you don't see the A/B Test option in your Single Send campaign yet, it hasn't reached your account yet. When A/B testing is available in your account, we'll send you an email letting you know! The information in this article is subject to change.
Table of Contents
- Overview
- Set up your A/B test
- What happens after you publish
- What to expect in your report
- Common Questions
This article covers A/B testing for Single Send email campaigns only. The settings described here don't apply to any other type of email campaign.
Overview
A/B testing lets you send two versions of your Single Send email, Version A and Version B, to a small part of your audience first. After the test runs, the version that performed better is sent to everyone else in your audience automatically.
Use an A/B test when you're not sure which message will connect best with your supporters. You might compare two subject lines for a year-end appeal, two different email designs for an event promotion, or two different senders for a donor thank-you.
Here's how it works:
- You choose which metric decides the winner: open rate, click rate, or conversions.
- You choose how much of your audience gets the test, from 10% to 50%. The test group is split evenly between Version A and Version B.
- You choose how long the test runs, from 1 hour to 24 hours.
- When the test ends, the two versions are compared, and the winner is sent to the rest of your audience.
For example, say your audience has 10,000 people and you set the test audience to 20%. Feathr sends Version A to 1,000 people and Version B to 1,000 people. When the test ends, the winning version goes to the remaining 8,000 people.
New to Single Send campaigns? Start with How to Publish a Single Send Campaign. This article only covers the A/B testing steps.
Set up your A/B test
Turn on A/B testing
Step 1: Click +Add Campaign from the Campaign table or from a Flight, select Single Send Campaign, and give your campaign a name.
Step 2: In the first step of the campaign wizard, find the A/B Test option and turn the toggle on.
When the toggle is on, the A/B test settings appear below it. When the toggle is off, the first step looks the way it always has and your campaign sends one email to your whole audience.
Choose your test settings
Tip: Before you start setting up your A/B test, take a moment think about what you want to learn and how your winning metric will measure it. A clear plan leads to clear results. If Version A and Version B differ in lots of ways, it can be hard to tell which change made the difference.
Step 1: Choose your Winning metric. You decide which metric matters most for this test. The metric you pick decides which version wins, and the winning version is the one that gets sent to the rest of your audience.
| Winning metric | The winner is the version with… | Good for testing… |
|---|---|---|
| Open rate | A higher open rate | Subject lines and preview text |
| Click rate | A higher click rate | Email content, like images, buttons, and wording |
| Conversions | More conversions toward your campaign goal | Which version leads to more donations, registrations, or other goals |
Step 2: Choose your Fallback winner, either Version A or Version B. The fallback winner is only used if both versions finish the test with exactly the same result.
Step 3: Use the Test audience percentage slider to choose how much of your audience gets the test. You can pick 10%, 20%, 30%, 40%, or 50%. The test group is split evenly between Version A and Version B, and everyone else gets the winning version.
Step 4: Use the Test duration slider to choose how long the test runs before a winner is chosen. You can pick 1, 4, 8, 12, or 24 hours.
Set up Version A
With A/B testing turned on, the Email Information section has a Version A tab and a Version B tab. You'll set up Version A first.
Step 1: On the Version A tab, give the version a Version name (optional). The name helps you tell the two versions apart.
Step 2: Select a template. Once a template is selected, you can click Edit content to make changes to it, or Change template to pick a different one. Learn how to build an email template here.
Step 3: Choose a verified From email address (required). To use the name and mailing address saved with that email address, leave Use contact info from verified email address checked. Learn how to verify an email address here.
Step 4: Add an Email subject (required). You can also add a merge tag to personalize the subject line, and Preview text (optional), which some email clients show next to the subject line. You'll need to select a template before you can add a subject line or preview text.
Step 5: Click Save as draft at the bottom right of the page.
Important: Save your campaign before you move on to Version B. If you haven't saved yet, the Version B tab shows the message "Save your campaign to start editing Version B," and you won't be able to edit it until you save.
Set up Version B
Step 1: Click the Version B tab.
Step 2: Choose how to start Version B:
- Start from scratch. Fill in each field manually by following the same steps you took to set up Version A.
- Start from a copy of Version A. Click Copy from Version A, then click Copy to confirm. This copies everything from Version A to Version B except the version name. Copying overwrites anything you've already entered in Version B. If you change your mind, click Cancel in the confirmation pop-up.
Step 3: Give Version B its own Version name (optional). The name helps you tell the two versions apart.
Step 4: Make the changes you want to test. After copying, you can still edit every field in Version B:
- Testing the email design or content? When you copy Version A, Feathr creates a copy of Version A's template for Version B, so you can click Edit content and change it without affecting Version A. To use an entirely different template, click Change template.
- Testing the subject line or preview text? Update those fields and leave the template as it is.
- Testing the sender? Choose a different From email address.
Step 5: Click Save as draft.
Publish your campaign
Finish the rest of the campaign wizard the same way you would for any Single Send campaign: set your Goals, send a Test, choose your Audience, then review and Publish. See every step in How to Publish a Single Send Campaign.
A few things to keep in mind for an A/B test:
- Goals: If you chose Conversions as your winning metric, you'll need to add a goal before you can publish. Any goal type works, and if you add more than one, a conversion toward any of them counts. Learn more about goals here.
- Test: Choose which version to send as a test: Both versions (the default), Version A, or Version B. With both versions, each test recipient gets two emails, one per version. The subject lines start with "[Version A]" or "[Version B]" so you can tell them apart.
- Audience: A/B tests work best with larger audiences. With a small audience, each version only reaches a handful of people, so one or two opens can decide the winner. If each version of your test would go to fewer than 100 people, you'll see a warning, but it won't stop you from publishing.
- Send time: Your send time is when the test starts, not when your whole audience gets the email. The winning version sends after the test duration ends.
What happens after you publish
Once your campaign sends, here's what happens:
- The test starts. Feathr sends Version A and Version B to your test audience, split evenly. Each person in the test gets only one version.
- The winner is chosen. When the test duration ends, the two versions are compared using your winning metric. The version with the higher result wins. If the results are exactly tied, your fallback winner wins.
- The winner sends. Feathr sends the winning version to the rest of your audience automatically. You don't need to do anything.
The winner is chosen based on which version has the higher number. It doesn't wait for a certain amount of data or check whether the difference is large, so even a small lead decides the winner.
What to expect in your report
When A/B testing is on, your Single Send campaign report has two tabs: Report and A/B test. Open the A/B test tab to see how Version A and Version B compare.
The A/B test tab has four parts:
- Version cards: A preview and the details of each version's email. The winning version has a Winner badge.
- Test summary: How many emails each version sent, and each version's open, click, and conversion rates.
- Performance over time: A chart comparing both versions' open, click, and conversion rates during the test.
- Version A and Version B: Each version's sends, deliveries, opens, and clicks over time.
Until your test emails send, the Test summary, Performance over time, and Version sections show No data available. Don't worry, that's normal! Your results will show up once your test starts, so just check back a little later.
Note: The Performance over time chart covers the test period. Keep this in mind when you compare it to the Report tab, which includes everyone who received the winning version after the test ended.
Common Questions
Which campaign types can I A/B test?
A/B testing is available for Single Send email campaigns only. It isn't available for Auto Send or Drip campaigns.
Why can't I edit Version B?
You need to save your campaign first. Click Save as draft at the bottom right of the page, then go back to the Version B tab.
What does Copy from Version A copy?
It copies every Version A field to Version B except the version name. That includes the template, sender, subject line, and preview text. Feathr makes a separate copy of the template for Version B, so editing Version B's template won't change Version A. Copying overwrites anything already in Version B.
Which winning metric should I choose?
Pick the metric that matches what you're testing:
- Testing subject lines or preview text? Choose Open rate, since those are what people see before they open.
- Testing the content inside the email? Choose Click rate.
- Want the version that drives more donations, registrations, or other goals? Choose Conversions, and make sure your campaign has a goal.
How is the winner chosen?
When the test duration ends, the version with the higher result for your winning metric wins, even if the lead is small. Your fallback winner is only used if the two versions are exactly tied.
Can I use Conversions if my campaign doesn't have a goal?
No. Feathr needs a goal to count conversions. Add a goal in the Goals step of the campaign wizard, otherwise you won't be able to publish. Learn more about goals here.
How big does my audience need to be?
There's no minimum, so you can still publish with a small audience. But Feathr shows a warning when each version of your test would go to fewer than 100 people.
When will everyone in my audience get the email?
Your test audience gets the email at your send time. Everyone else gets the winning version once the test duration ends. For example, with a 12-hour test, the rest of your audience gets the winning version about 12 hours after your send time.
Can I change my A/B test settings after I publish?
Only while your campaign is scheduled and hasn't started sending. Once sending starts, your A/B test settings are locked, and you can only edit your Goals.
What happens if I duplicate an A/B test campaign?
The copy keeps your A/B test settings and Version B, and Version B gets its own copy of its template. Test results start fresh, and the original campaign's winning send isn't copied.