Testing Faster, Testing Better

In the midst of a pressure-cooker economy, circulators are re-engineering their testing strategies to drive results deeper, faster and more efficiently.

By Barbara Love
Wednesday, April 30, 2008

---

We’re warning you up front: You’re going to read the word “test” ad nauseum in this story. But with so much change in the economy and the industry, testing is the only reliable way to know how to move forward with marketing plans. The ageless mantra “Test, Test, Test” and the thoroughness it implies is quickly being refined to a point where publishers are creating and performing tests in more targeted and strategic ways. Testing itself has changed. Online testing has become easier to implement, which has also opened up new and exciting cross-platform tactics. And multivariable testing, an intricate but highly illuminating technique, is gaining mainstream traction.

What’s Being Tested

Gordon Bell

Gordon Bell, president of LucidView, who works with a number of publishers on marketing tests, reports a great deal of price and offer testing and variations on presentation of the offer (price per copy, number of issues free or strike through pricing).

A big change has been in the depth and efficiency of testing, he explains. Marketers have always tested new prices, offers and creative packages, but there’s a clear advantage in going beyond what everyone else is doing. For one thing, “offer” is not so simple as “$12 for 12 issues.” What is the real value and how is it presented? Is a “Just $1 an issue” dot, or “Save 74 percent off the newsstand price” highlight helpful? With a buckslip listing articles and features, or copy positioning the magazine as a “must read,” can you support a higher price point or encourage a longer subscription term? If a new package proves successful, then break it apart.

Where the Opportunities Are

David Ball

David Ball, VP, consumer marketing, Meredith Publishing, sees testing opportunities almost everywhere—in all sources. “We see opportunities in most sources,” he says. “Online gets us quicker results, but we’re always seeing significant breakthroughs in our direct mail source. And we like to be creative with our agents.”

But the testing itself can be far more sophisticated than the traditional AB split, which still prevails. Bell says one big opportunity is more strategic testing—planning a series of tests over the next 6-18 months that build upon each other to speed learning and growth. For example, a large creative test of 10-15 elements can quickly identify what makes a difference. This can be followed by smaller creative tests leveraging these insights to optimize important elements
Another opportunity, especially for renewals, is multi-channel testing. “Use email and a special landing page to support the direct mail effort,” Bell advises. “Coordinate the timing of emails, direct mail, and telemarketing.”

Improved Technology and Methodology

Luckily for all circulation marketers, the methodology and technology surrounding testing has improved. Although the science of testing has evolved over the last 80 years, only in the last few years have these complex statistics reached mainstream marketers. This allows marketers to move far beyond the basics of split-run testing.

And the technology in many areas related to testing—database marketing, use of prospecting and retention models, list optimization and co-op databases—has become more affordable. In addition to doing tests on creative, price and lists, publishers are doing a lot of work on database regression modeling and other modeling techniques.

From the technical aspect, there are new companies out there that have shortened the amount of time it takes to create models. That allows publishers to do more modeling and refresh the models more often.

This is particularly important in the online side, where publishers are in the market every week with information. It used to take several days to get reports on what’s happening on the Internet. Now the turnaround time is a few hours. The faster turnaround time allows publishers to target people much more accurately.

Testing What Will Move the Needle the Most

Of course the most excitement in testing is not in tweaking elements, but in finding the big breakthrough that really improves profitability.

The Financial Times, which has a U.S. circulation of 152,000, focuses its testing by source on direct mail, the Web, FSIs, billing and renewals. “We test lists, offer and creative primarily, because that’s where the lift is,” says Richard Varey, VP, consumer marketing.

In the spring of 2007, Varey tested adding a graphic of the newspaper. A call-to-action URL, new benefits copy, requesting email address, new layouts, a lift letter, a live stamp and envelope teasers were also tested. The first three produced a 42 percent lift in net response.

Test Frequently, But Don’t Over-Do It

Beth Ifcher

Beth Ifcher, Hearst’s VP, consumer marketing director, says 10 to 15 percent of any campaign budget is normally allocated to testing, and is trying not to over-test. “We have to decide if we are looking for short-term or long-term solutions, and we have to prioritize,” she says. “There’s a point in time where if you do too many tests the results may not replicate the role-out.”

Patrick Hainault, consumer marketing director at Mansueto Ventures, is also careful not to over-test. When Mansueto acquired Inc. and Fast Company from Gruner + Jahr in 2005, for example, it devoted “an inordinate proportion” of its budget to testing—nearly 60 percent. The company has now settled for 20 percent or so in all channels. “The initial over-testing was by design. There was a serious lack of fresh, trustworthy data on which to base decisions. As we filled that knowledge need, we began reducing our tests,” he says.

While the requirements at Mansueto are to push the envelope, Hainault will only spend on testing if he foresees a real upside. “We’ve agreed to use our judgment and eliminate tests about which no one is really excited,” he says. “Unless you mail tens of millions of yearly pieces, futile tests are those that only play on the margins—one free issue instead of two, matched label instead of a window, rounded price, and others. If they weren’t exciting enough to test the first three times you considered them, it makes little sense to later test them just because there’s still room in the budget.”

Meredith puts tests through a rigorous planning process, particularly for direct mail. The manager proposing a test must present the rationale for the proposed test, show if and where the proposed idea is currently being used in the marketplace, and present a breakeven P&L to demonstrate how much of a lift a test will need to win. The senior staff then evaluates the potential ROI versus other tests Meredith could be doing.

For cost-effective testing, Varey at Financial Times relies on “risk assessment based on experience, noting sector response trends and market conditions, and using refined modeling.”

Accordingly, a price elasticity test Varey developed resulted in a price that produced a better long-term ROI. After many tests yielded unitary results, the magazine tried $49 for a six-month subscription. This outperformed $149 for 12 months by a factor of 4.5. The new price has been rolled out.

Multivariable Testing: A Major Leap Beyond Split-Runs

Multivariable testing (MVT)—allowing marketers to test more variables more efficiently—is a hot topic of conversation among circulation marketers nowadays, but it has yet to be widely adopted.

Almost all testing by publishers is in AB splits, says Bell. “Most people in publishing have heard of multivariable testing but nine out of ten publishers have probably never done it,” he says.

MVT requires a large universe, costs a lot more than a split-run test, and requires some expertise that is not present in most publishing companies. But with multivariable testing, a circulator can gain information much more quickly, and more cost-effectively than running a long, drawn out series of AB tests.

“So much of it comes down to the risk,” says Bell. “It’s not a step beyond split-run tests, it’s a major leap. You can clearly see the results and clearly quantify how profitable the test was.”

Many of Bell’s clients (who include publishers, packaged goods companies and retailers) have tested up to 20 variables, maybe more. In theory, there is no limit.

MVT can offer results and insight far beyond AB splits. And while this method of testing can be more involved and more expensive, like any other form of testing, successful results can pay for the effort many times over. The Financial Times conducted its first MVT program in direct mail with LucidView in the spring of 2007. There were two test matrices—one for creative and one for price tests, Varey explains. The creative matrix consisted of 12 panels or recipes, with 11 test elements, the price matrix four elements. With these elements, the publication tested 2,064 combinations.

“If we used split-run techniques, not one effect would be statistically significant and the interactions would be impossible to see,” says Varey. “Many elements would not warrant a test panel on their own. The program accelerated our learning and will pay for itself on rollout,” he added.
 

Internet Tests Are a Different Ballgame

Just by virtue of being the younger medium and rising materials costs, the Internet is a big focus for testing right now. Plus, for many publishers, online is the fastest growing segment.

“Because the cost of testing variations on a theme is so much cheaper on the Web, we tend to have a little more fun with it,” says Hainault. “We allow ourselves to get more granular in what we observe.”

Using Epsilon’s ChannelView tool, Financial Times is able to track mail prospects that respond via other sources, particularly online. “Online efforts have less time and space to attract prospects,” says Varey, “at least until they get to the landing page, so the focus is on offer and brand.”

Hainault has used Web-based tests for premium selection, term presentation and price points to guide postal efforts. “If this means accepting a tradeoff between purity and alacrity,” he says. “I’ll take alacrity any day.”

Alacrity, says Hainault, because he can always back-test. For example, if an “unpure” test indicates an element is a winner you have two choices: Rolling it out right away or building a direct mail test for further proof.

“By rolling with it right away, you run the risk that it may have been the wrong decision, but you also give up the better than 50-50 chance that it might help response now,” says Hainault. “Last year, when we were still using premiums, we tried this approach. We tested a whole bunch of premiums online and then rolled out with the winner, back-testing against the old control. Results held. Trying to get to the same point in a textbook direct mail fashion would have eaten a lot of our testing budget and might have required a few mailings. Opportunity costs aside, that’s just too much of an investment.”
Varey at Financial Times has found that continuous service offers do better online than in print, while shorter-term beats longer in print. The online audience for FT skews younger than print, he notes, but quality in terms of demographic and lifetime value is comparable. 

Testing methodology has come a long way on the Web, says Bell. The first online platforms for split-run and multivariable testing have only been around for five or six years. These platforms, used by Conde Nast and the Financial Times, among others, make it easier for companies to create all the different versions of Web pages to test. And while these platforms are primarily designed to take advantage of multivariable testing, they are most commonly used by publishers for AB splits.  

6 Tips to Test…for Tests

Keep these guidelines handy when you’re about to begin...well, you know:

1. Make sure you know what your competitors are doing. You might think you can get some ideas there, but just because a package or a promotion is a competitor’s control, don’t assume it will work for you.

2. Look at other media, not just magazines. While sweepstakes haven’t been used by magazines in a while, Meredith is looking at them again because they are working in other businesses and on the Web.

3. Test when the economy and the competition are tough. During these times people tend to cut back on marketing and testing. Even though that might be a safe route for the present, it doesn’t help you in the future. Continue testing, but try to do it more cost-effectively.

4. Don’t ignore tests on creative elements or segment-specific tests. “Everyone focuses on price and offer,” says LucidView’s Bell. “But where the opportunity often lies is where the competition isn’t looking. Sometimes a few small creative changes can lead to a better response and better profitability.”

5. Don’t test tweaks. Make sure you are testing things that have the potential to improve profitability.

6. Don’t wait for your campaign to fatigue before you test. Stay ahead of the curve.

Multivariable Testing: An Illustration

By Gordon H. Bell, President, LucidView

Scientific multivariable testing offers the power and freedom to test numerous marketing elements, all at once, with greater accuracy, deeper insights, yet with the same sample size as an A/B split. Here’s an example of how multivariable testing can be deployed, with illustrated results that reveal the level of detail this testing format can provide.

One marketing team identified 16 ideas for increasing response in their direct mail acquisitions program. With guidance, they split these ideas into two multivariable tests, focusing on creative and price/offer elements. The first test of 11 creative elements—including changes to the envelope, letter, inserts, and order form—pinpointed 6 significant effects for a 55 percent jump in response (using 1/7th the sample size required for split-run testing).

The second test focused on the five price and offer elements shown below.

Eight test packages were mailed, each with a specific combination of all five elements, as defined by the scientific test design (not shown). Analyzing response rate across all eight packages, results were summarized in the bar chart below, showing four significant effects:
• The low subscription price (A+: $12.99 annual price) increased average response 1.33 percentage points over the higher $19.97 price (a 23 percent lift)
• The longer subscription term (B+: 18 months) reduced response, on average, 0.69 percentage points (12%).
• Showing a specific expiration date (E+: “offer expires 6/19/03”) increased response 0.380 percentage points (6.55%).
• The non-significant effects were also valuable. The 2- and 3-year options (D+) did not hurt response and led to a few long-term subscribers, so the team decided to keep the checkboxes. Showing the cover price (C+) had no impact, so it was removed to leave room for value-added copy.

Interactions—impossible to see in split-run testing—are particularly valuable in price and offer tests. This test also quantified one significant interaction, shown below.

Moving from left to right on the lines, you see that a lower subscription price increases response. Moving from the orange line (at bottom) to the green line, you see that, on average, a 1-year subscription term leads to higher response than an 18-month term (tested at the same per-issue price). However, the two points at the upper-right show that if subscription price is kept low, then there is no significant difference in response between a 1-year and 18-month subscription. Therefore, the circulation team can lock in subscribers for a 50 percent longer term without hurting response.

This interaction not only uncovered a way to lock in subscribers for a 50 percent longer term without hurting response, but also provided greater accuracy in assessing all combinations of subscription price and term. Both of these benefits are impossible to see with split-run testing.

Scientific multivariable testing offers more freedom, yet requires greater discipline in planning, launching, and analyzing each test. Interestingly, if these same elements were tested with A/B splits:
• Only 2 of the 10 important effects would have been significant
• For equal confidence, sample size would need to be over 800 percent greater
• Interactions would be impossible to quantify