When Apple introduced iOS7 last year people were wowed by its stark design and, of course, the new features. Unfortunately for some users, it was also making them sick. Specifically, some users reported headaches and nausea brought on by using the zoom and slide functionality and the floating icons above animating wallpaper.
TidePool mobile app developer Jenni Leder described it to The Guardian this way. "It's not apps that affect me, but accessing them. Tap a folder and the view zooms in. Tap an app and it's like flying through the icon and landing in that app's micro world -- and I'm getting dizzy on the journey there." This wasn't a case of people grousing. As The Guardian noted, 3D effects can evoke these symptoms for people with vestibular disorders.
Apple typically plays its cards close, but we can assume it tested iOS7 numerous times and in several different scenarios. So this example is particularly illustrative: If Apple can't run a perfect test of a new app how can your average everyday company?
Doing it right
The literature on this subject -- which is particularly rich and in-depth regarding multivariate and A/B testing for email and Web content --is discouragingly rife with all the ways such a test can go wrong. Perhaps the sample size was too small or there was an internal event that changed visitor behavior. Other variables to watch out for as you run these tests over and over, experts say, are slight changes in technical environments or measurement tools or even incoming traffic or traffic mix.
But the rewards of successful multivariate testing are well worth it. And it isn't rocket science either, says John McDougall, author of the Web Marketing On All Cylinders and president of McDougall interactive.
He told CITEworld about tinkering with the landing pages for a lead generation campaign for its IT support client. The landing pages were built using Unbounce, a hosted platform for landing page creation and testing. When they started, the cost per lead (CPL) was around $4.
McDougall began A/B testing the landing pages first by experimenting with headlines.
"Amazingly, simple changes to our headlines where we were more specific and to the point as to what our services would do for the consumer, led to a substantial improvement in conversion rates and our CPL."
The company also A/B tested the pay-per-click copy, with similar results, McDougall said. "For our PPC ads, we found simple changes to our copy, again being a little aggressive in the bottom line benefit statements of our service, helped to boost conversions at the very top of the funnel." Taken together, he said, these efforts shaved about $2 off of the CPL on a per lead basis.
Sign up for CIO Asia eNewsletters.