A/B test campaign messages with Campaign Composer

Updated June 13, 2024 17:45

The key to unlocking high-performing campaign messages is frequent, small-scale A/B testing, for both message content and message timing. We recommend always including at least 3,000 subscribers in each A/B test campaign you run to ensure optimal test results.

Testing new approaches in your messaging program will help you uncover important preferences and leverage points for your subscribers, including:

The best images and media to use
The kind of urgency language that drives the most conversions
When your subscribers are most likely to not only open your messages but click through and complete purchases
How to best structure messages that retarget subscribers who didn’t click or complete a purchase after your initial message

You can A/B test both text and email messages in the same campaign.

Before you get started

Make sure to check out our best practices for testing campaign messages.

How do A/B test messages work in a campaign?

A/B test messages work by first sending each campaign message variation to a percentage of your total audience for the campaign. As recipients interact with your campaign messages, the performance of each variation is tracked. Once a campaign variant with the best performance is identified, the winning variant is sent to the remaining audience of the campaign.

When setting up your A/B test campaign, you can customize:

What to test: Send times or message content, including copy, links, and media
Test details: Use the slider to set the percentage of total recipients you’d like to send test variations to (default is 20%). After the selected amount of time (default is 2 hours), the best-performing variation is identified based on the selected criteria (default is click-through rate), and your remaining recipients will receive that winning variation.

If you wish to send the variations to all of your recipients at the scheduled time, adjust the slider to 100%.
Winning criteria: How to determine the winner of your A/B message test. For more, check out Campaign performance and metrics.
Test duration: How long to run the test before determining a winning variant and sending it to the remaining audience recipients.

A/B test for message content

Note that you can A/B test either message content or message send time for each A/B test message.

Go to Campaigns and click + Create campaign.
Click A/B test.

Notes:

You can convert a campaign text message to a A/B test for message content by clicking + Add A/B test variation while creating a message.

Subsequent messages can be additional A/B tests, texts, or email messages, but you can’t change the initial message type once selected.
For your A/B test message, select from:
- What channel you want to test: Text message or email
- What you want to test: Send time
  
  Once you’re ready, click Next.
Next, set up your content A/B test message. Complete the following:
- Campaign name: Optional, but recommended. Click the Pencil at the top of the page to edit the internal name of the campaign.
- Message name: Optional, but recommended. Enter an internal name for the first message of your campaign.
- Tags: Click Add tags and select an existing tag or type a name to create a new tag. You can add up to five per campaign message. Learn more about campaign message tags.
- Audience: Click Edit and select one or more segments of subscribers to receive your campaign. We recommend including segments that include at least 3,000 subscribers for each A/B test. You can also click Exclude to further refine the audience of your campaign. You can see an estimate of message recipients for your message above the Done button.
  
  You can also change Smart Sending settings (not recommended) from the Audience side panel.
- When to send: The date and time to send your first campaign message. For time, you can choose between the time of your brand’s HQ (chosen in Settings), or based on the time zone of each subscriber.
- Test details: Set the percentage of recipients who should receive the test, the winning criteria, and the test duration. For more on test details, see above.
  
  Once you’re finished, click Save.
Create two more message content variations to test. For each content variation, click Add, and then design your variant message. You can include images, GIFs, links, and more.

If you’re testing text message variations, check out Design your campaign text message and Best practices: Campaign message design.

If you’re testing email message variations, check out Design your campaign email.
By default, A/B tests include two variations. Click + Create another variation to add another. You can add up to 30 variations.
If you want to add additional messages, including additional A/B test message, click + Add message, choose the message type, and repeat the above steps.

For more on non-test messages, see Create a campaign.
As you add and complete each campaign message, make sure to test each message to make sure everything looks right. Click Send test message, enter your mobile number or email address, and then click Send test.
Once you’re finished adding messages to your campaign, click Review & schedule campaign.

Once you schedule your campaign, you can find it on the Campaigns page by filtering for Scheduled campaigns.

A/B test for message send times

The steps to A/B test for message send times are similar to testing for message content with two key exceptions:

When you first add a new A/B test message, select Send time instead of Content.
For Test details & send time, click Edit and then complete the following:
- Time zone: Choose between the time of your brand’s HQ (chosen in Settings), or the time zone of each subscriber.
- Send time 1: The first send time variation for your message.
- Send time 2: The second time variation for your message.
  
  Click + Add variation to test additional send times. When you’re finished, click Save.
Continue setting up your campaign variations. The process is similar to setting up content A/B test variations.
If your message variations include links, you must select a Test parameter for links. A dynamic value is automatically assigned to the parameter, which you can use to identify message variations in third-party analytics platforms, like Adobe Analytics and Google Analytics.
Notes:
- You can choose the parameter you want to use to identify variants, but you can’t change the parameter’s value. Attentive assigns a dynamic value, which overrides any values already set for that variable.
- This chosen test parameter is also used for links in all message variations.
Example: For a send time A/B test campaign named ‘2024 Summer Sale,’ you send one message variation on 7/1/2024 at 10:00 AM CT and another at 2:00 PM CT. You choose utm_campaign as the Test parameter. The value for the first variation is Sale-announcement-2024-07-01-10-00-AM and for the second variation is Sale-announcement-2024-07-01-02-00-PM.

A/B test for email design and subject lines

You can A/B test both email subject lines and email designs. This can help you learn which strategies drive higher engagement and ROI. For more information, see A/B test email campaign messages with Campaign Composer.

View A/B test results

To see more information about how each variation performed in an A/B test, hover over the info icon in the Winner label for the variation that won to see a popup with detailed information.

The popup that appears when you hover over the info icon in the Winner
label for a winning variation in a campaign A/B test, which explains what was
tested, when the test was run, and the performance of each variation of the test.

The popup provides the details of the A/B test, including what was tested (e.g., click-through rate, unsubscribe rate), when the test was run, and the performance of each variation of the test.