Hearst Experiments With Multivariable Testing

After successful multivariable testing on direct mail campaigns for Food Network Magazine, Hearst throws customer data into the mix.

By Chandra Johnson-Greene
Monday, May 3, 2010

Although multivariable testing has been around for a few years now, publishers that use it regularly are still considered by some to be early adopters. Whether the reason is cost, time or the reluctance to do away with the traditional split-test run, many companies are still hesitant to experiment with multivariable testing. Hearst, however, has been doing more multivariable testing, adding the process to its direct mail packages, and is also getting ready to combine it with the information it has in its database.

When asked if it was upper management that was skeptical about doing multivariable testing at the beginning, promotions director Heather Plant admitted that she was, in fact, the skeptical one. “My concern was that it was going to get too complicated,” she says. “Because when you’re doing a campaign, you could be testing as many as 20 elements at once. Each panel may contain eight of those elements, so executing the test does take extra time.”

Last year, Plant and her team focused on creating the most optimal package possible for Food Network Magazine. They had a control package already created for the new launch, but wanted to improve on what they had done, so they decided to run a multivariable test. “First we did creative price and premium testing,” she says. “Some of the things we tested included using a window envelope versus a closed envelope, adding photos of Food Network stars or using descriptive copy on the order form, adding lift notes, removing the brochure and changing some copy.”

Through multivariable testing, the team was able to find out, for example, that control copy, which focused on what was inside the magazine, worked better on the order form. “Multivariable testing gave us the opportunity to test things that we normally don’t have the time or money to test when doing an A/B split test,” she says. “For example, changing the copy on an order form is something that we would have never slated for an A/B test. And we were able to test 25 elements with a sample size of 600,000 pieces. With an A/B split, we would have had to test 4.7 million pieces in order to get all of the combinations tested.”

And while Hearst chose to use a vendor to assist with the testing, the uplift in responses the campaigns received more than paid for it. “We were able to get our control campaign out three campaigns earlier than we would have by doing A/B testing,” says Plant. “And with a 10 percent lift in response, we gained over $150,000 in net revenue, which more than paid for the testing.”

The Next Level
While doing a multivariable test on direct mail packages involves testing the creative elements, what would response rates look like if behavioral and transactional data were thrown into the mix? That is what Charles Swift, Hearst’s VP, database strategy and marketing, is getting ready to find out.

Swift is currently working with his team to build a system that will screen and segment customers to find out what their likelihood to renew their subscriptions to Hearst’s products will be. “We’re trying to set things up where, for example, we can say ‘Here’s a population that has subscribed to a product and there’s a 20 percent chance they’ll renew,’” he says. “We’re then going to try and figure out how we can change the predicted behavior. So we’ll use multivariable testing to find the right messaging, price point and hook to bring them in. And for the customers that have a higher percent chance of renewing, we can test to see if we can stop giving them the lower rate the next time around or if we can avoid giving them as many premiums.”

Swifts adds that he may also use multivariable testing to find out which customers should be contacted to renew six months ahead versus those who might only need to be contacted two months ahead. But he stresses that multivariable testing doesn’t make a team’s creative ideas better. “What it does is it gets you the best idea that you have faster,” he says. “Some get frustrated when they do the testing and response rates are still low, but if you test 20 ideas and all of them are bad, then you’re not going to get much out of it.”

Online Testing
New software options are allowing publishers to take multivariable testing online. Mark Wachen, founder and former managing director of Autonomy Optimost, a provider of multivariate testing software, discussed at a recent industry event the concept of what the company calls “adaptive targeting,” or capturing data such as geography, behavior and demographics in addition to the standard test elements, similar to what Swift is planning to do for Hearst’s direct mail.

With adaptive targeting, according to Wachen, America’s Test Kitchen was able to test different subscription price points against geographic regions. The company found that audiences living in the West were less sensitive to price changes while those in the South were more sensitive.

But online multivariable testing can be a bit challenging, especially for publishers unfamiliar with the process. “The possible sources of variation can be hard to pin down,” says Gordon Bell, president, LucidView. “You can see opposite results from week-to-week depending on if a different promotion was being used, if a partner of yours is promoting your magazine differently than you are, and if the search engine results that week have changed.”

But despite the challenges, Gordon adds, the results both offline and online can be worth the effort. “Multivariable testing takes time, effort and cost,” he says. “But that being said, the overall return on testing will be greater and faster than doing A/B testing over time—greater risk equals greater reward.”