It's no secret that Microsoft's Bing search engine doesn't get much love compared to Google. So to help poor Bing snag a few more users, Microsoft set out to show that its search product was just as good—if not better—than big bad Google. Given the chance in a Pepsi Challenge-styled blind test, Microsoft says, people actually prefer Bing over Google.
And so was born the Bing It On campaign, based on a Microsoft study of "nearly 1,000 participants" who ended up preferring Bing over Google two-to-one in blind tests. To drive home its message, Microsoft embarked on a major advertising campaign and set-up a website called BingItOn.com where anyone could take the Bing challenge.
Microsoft's claims were entertaining, but they didn't sit well with Ian Ayres, an economist and professor of law at Yale. On Tuesday, the good professor cried shenanigans against Microsoft.
"The 2-1 claim seemed implausible," Ayres wrote on the Freakonomics blog. "I would have thought...it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1."
To find out if he was right, Ayres tested 1,000 Mechanical Turk workers—a similar sample size of Microsoft's original Bing study—to see if he could duplicate Microsoft's results using the Bing It On site.
Binging it on
As any scientist in the budding (and totally made up) field of "searchology" knows, the results of any experiment cannot be taken as fact unless they can be duplicated. So can we say with absolute fact that people prefer Bing or Google? Do we finally have enough data to create the Big Bing Theory?
Not according to Ayres, who found that most people, when left to their own devices, actually prefer Google over Bing. He did find that when people searched for a Bing-suggested search term on the Bing It On site, preferences for Bing and Google results were about even. But when people could use their own search terms or use terms pulled from a list of the Web's most popular searches... it was the big "G" all the way.
Ayres argues that the second part of his study "indicate[s] that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results."
Ayres also suggests throughout the blog post that Microsoft has been misleading with its Bing It On ads. The problem, says Ayres, is that Microsoft has not released the results of the more than 5 million people who have taken the Bing challenge on BingItOn.com to see if they line up with Microsoft's original study.
Sign up for CIO Asia eNewsletters.