DMA13 Session Highlight: Never Trust Anything, Test Everything

This summary is a guest blog post from Ryan Phelan, VP of Strategy Services for Digital Impact/Acxiom, and a speaker in the Email Marketing Post Conference Workshop at DMA13.

ACXIOM_ryanphelan-headshot-closeup_2013 (2)Nothing anyone in marketing tells you is true.  Shocked?  Well, you should be.  It’s the mantra of the strategist to trust nothing and test everything (ok, maybe not everything).  Why?  Because  every company is different.  Every company sells different products and has different customers.

What’s even more surprising is that one of the top challenges is the ability to test.   It’s based in time, resources and money but it also involves the marketer not knowing how to test.   It’s tough because the average marketer struggles to get email campaigns out the door and testing takes time.  You have to be focused and look at the item you want to test, come up with a hypothesis and test it accurately.  Accuracy matters.  You must make sure that you test something accurately so that you don’t end up drawing the wrong conclusion.

Think about this.  A marketer wants to test a Call To Action of “Buy Now” and “Learn More”.   They have two pieces of creative and because of the size of the font, the “Learn More” button takes up two lines and makes the button bigger.  The marketer runs the test and “Learn More” creative is determined to be the winner.  Did “Learn More” really win or did a bigger button actually win?  You would not know because the test you ran was invalid.  The variables have to be equal in each creative.  At a basic level, this means audiences have to be equal in makeup (activity, join date, purchase and other factors), creative, message and timing.

So, how do you run an accurate test?  There are multiple kinds of testing.

A|B Testing

  • Based on the principle of 2 identical pieces of creative that have one variable changed in one creative
  • PROS: Most commonly used and effective
  • CONS: can take a long time to ascertain a direction for multiple elements

Multi-variant testing

  • Based on the principle of multiple pieces of creative with multiple variables changed in one creative
  • PROS: can test multiple elements at one time
  • CONS: hard to execute in email based on the population sizes needed to gain a statistically valid sample of response

Local Control Groups

  • At a campaign level, groups of customers are not sent the creative to ascertain their behavior against groups that received the creative
  • PROS: Easy to implement at a campaign level
  • CONS: Those customers will not get the creative you have available

Universal Control Groups

  • At a program level, groups of customers that are reflective of the brand, that are not sent email campaigns for a longer period of time (3+ months) to ascertain their behavior against groups that received the campaigns
  • PROS: Powerful way to show the effectiveness of your program
  • CONS: Hard to get company buy in to isolate a valid sample of customers that would not get creative during the time period

So, how do you test?  Most marketers can accomplish the testing they need through A|B testing and as you get more advanced and skilled, you can move into other ways listed above, but the key step is to develop a plan.  Take a three month period and define the goal of testing for that period.  What are you going to test and what is the path that you should take.

For example, you want to test if your headline copy is truly effective.  You could test having it and not having it, but what if you had a plan over three months to really be sure.  You could test:

  • Font color
  • Font type
  • Placement
  • Absence
  • 1 line of copy
  • 2 lines of copy

The variations are large, but if you have a plan, you can get to a point where you feel comfortable saying “My headline copy should adhere to these standards”.

Testing enables the marketer to know that what they are hearing at conferences or reading in papers work for their customers.  Don’t just trust some random stat.  Make sure you test what you hear to make sure it works for you.

Your Comment