Online Multivariate Testing

A robust way to test your site’s performance on-the-fly.

By Bill Mickey
Monday, July 6, 2009

---

Imagine if you had a Web site that profiled more than 600,000 companies broken down into 67,000 product categories. With that kind of scale you’d certainly want to make sure your visitors are finding specific information quickly and easily. This is everyday reality for ThomasNet, a media company serving the industrial engineering market, and to make sure their customers are happy, the Web team pursues an aggressive multivariate testing program, continually optimizing their site to drive more conversions and meet specific performance benchmarks.

“You can’t manage what you can’t measure,” says Tom Marren, ThomasNet’s director of Web development, “and your Web analytics can only show you so much. They can’t show you what happens if you do this.”

To that end, Marren, through Autonomy’s Optimost platform, regularly uses multivariate testing tactics in live scenarios. “The basic premise behind testing is you want to improve the performance of your Web site. The classic example is the e-tail shopping cart experience. Most companies want to improve their order rate and checkout, and the best way to do that is if you have a live site to test. You can do it in a real-world environment and get results quickly.”

It’s not just about shopping, either, though a subscription offer landing page and subsequent conversion is one easy parallel. Content publishers can also test navigational elements, registration funnels, newsletter subscription pages, and the arrangement of content on a page, among many other elements.

The benefit of multivariate testing over classic A|B split testing is you can test many elements at the same time and your time to completion is shorter than if you did a succession of A|B tests. “Multivariate testing leverages some very powerful statistics and modeling and can test a far greater range of options than A|B,” says Marren.
Automony manages the testing process for Marren, who identifies what elements need to be tested—“If it’s changing elements on a page, they can do it with their tools, if it’s a process we want tested, we have to give them the elements.”

Analysts then issue a report at the end of the cycle along with recommendations.

The Hardest Part: The Pretest Phase

“Test planning is the most important and often hardest part,” says Marren. Yet it’s this stage that will set the basis for success—“Otherwise you’re just throwing stuff at the wall. It’s about looking at your site metrics and understanding what’s going on and then doing a thorough test plan, which ultimately determines success. The rest is just execution.”

In conversion funnels, for example, Marren recommends testing on the inside and then backing your way out. In other words, start with the check-out step and work your way back to the review order page, then back to the shopping cart page, and so on. By working backwards, especially in a live environment, says Marren, you won’t “optimize an early step in a process only to drive traffic to one that is broken.”

One recent test, says Marren, was performed on company profile pages. The test examined what would happen if elements were rearranged and more color was added. Conversions—actions valuable to the supplier, such as clicking through to their Web site—spiked by five percent. That’s five percent on a page that gets millions of views per month, says Marren.

A Free Solution

ThomasNet’s Marren wouldn’t reveal how much they spend on the Optimost testing package, but Google’s Web Site Optimizer tool (Google.com/websiteoptimizer) offers multivariate and A|B testing features for free. As it turns out, Optimost is the technology behind Optimizer. “It compares very well,” says Marren, “but you don’t have access to analysts pulling the data and running reports.”

And with Optimizer, you’re relegated to testing more static elements. Dynamic data testing is not available. “Google Web Site Optimizer can probably test about 80 percent of what you want to test,” adds Marren. “For large database-driven Web sites like ours, it’s probably not a good fit.”