Sunday, March 27, 2011

The Purpose of the Control in Split-Testing Email Campaigns

Recently, I’ve been training a team member to take over the responsibility of managing email marketing. She has a degree in communications, and is a really strong writer. She took one look at the automated messages that set up in the reflex campaigns, and cringed.

She then proceeded to write new, much better copy to replace the old.
She wrote better subject lines, better headlines, better ‘calls to action’.

When we met and she showed me her plans, I complemented her on the great work she’d done. But then I recommended against replacing the old messages with the new ones.
When she asked why, I answered with a question of my own.

I asked her: “if you launch this change, and get a lift in response and sales, what will you do next?”

She looked confused. “next?”

I asked her “What happens if we double our response from these campaigns? What will we do next?”

From her frustrated expression, I could see that she wanted to say something like “celebrate?”, but felt instinctively that this wasn’t the answer I was looking for.

Her plan to roll out this new copy implicitly assumed that:
1. Her new copy was better (which it almost undoubtedly was, but even when I’m sure, I always look for proof)
2. Once we rolled out the new copy, we would have ‘fixed the problem’ represented by the old copy
3. Her work on this project could be crossed off her ‘to do’ list.

“How will we get to a 400% increase?” was my next question.

The best way I've found to continually move from good to better. Secure the ground you’ve gained, and use it as a jumping-off point to move forward again and again and again. To do this, you need to establish, and rigorously maintain a ‘control’.

And when does it end? Never. As they say in the business “Always Be Testing”.

The purpose of the "Control"

The “control” is your currently best-performing piece of creative for any single purpose. If you currently send out 3 different email messages to your customers like:
1. Welcome to my online store
2. Thanks for buying
3. Please buy from me again.

These three messages are your three ‘controls’.

If you have only one message that you’re using, and it’s performing terribly, and you know exactly what you need to do to fix it – it’s still your “control”. Everything else you do should be measured against it. You should not abandon it without making your new message challenge and defeat it. If your new message outperforms, then it becomes the new control.

Generally You conduct a split test by sending the ‘control’ to a randomly selected portion of your recipients, and the ‘challenger’ to the remainder. Typically, you would split your list into 2 equal groups, then compare the response of one against the other, and whichever performs best becomes the new ‘control’ ready to take on new challengers.

Choosing What to Test

There’s a reason why we were still running admittedly ‘awful’ email copy when my colleague was handed the assignment of taking over managing our email marketing. The reason: I was testing something I considered far more critical than the content of the message, and didn’t want to cloud the results, even though the existing copy ALSO made me cringe.

I was testing the timing of the message.

When I took on this project, I found that we were automatically sending out a reminder message to new customers encouraging them to make a purchase 30 days after they opened an account with us. I suspected that after 30 days, many of our customers might have forgotten they’d opened the account, and that we could boost response simply by mailing them sooner.

I started by cutting the time in half. A 30/15 day test. When that proved successful, I cut the time in half again to 15/7 day. And finally when the 7 day test won, I ran a 7/3 day test. Currently the 3 day message is my control. At that point I intuitively felt it was time to start testing other elements.
Here are a few suggestions on elements to test:
Subject line
Headline
Offer
Position of links
time limited versus open-ended offers
Personalized versus not