You've seen the Bing It On "blind taste test" search engine commercials in which users decide if they favor the results from Bing or from Google. But what would life be without drama?
When the Bing It On campaign launched in September of 2012, Bing claimed people prefer Bing 2:1 over Google. That was apparently too much for Yale Law professor Ian Ayres to swallow. In fact, it aggravated him so much that he dug into the claim. He wrote on Freakonomics, "I was slightly annoyed to learn that the 'nearly 2:1' claim is based on a study of just 1,000 participants. To be sure, I've often published studies with similarly small data sets, but it's a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I'm guessing is a low-six-figure study."
Ayres, along with four Yale Law School students, conducted their own Bing It On taste-test challenge before publishing their conclusion "A randomized experiment assessing the accuracy of Microsoft's 'Bing it on' challenge claims [pdf]."
"We found that, to the contrary of Microsoft's claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were 'ties')."
We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred. We randomly assigned participants to search for one of three kinds of keywords: Bing's suggested search terms, popular search terms, and self-suggested search terms. When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google). But when the subjects in the study suggested their own searches or used the web's most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing. These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results.
The group concluded, "Google has a colorable deceptive advertising claim against Microsoft."
Matt Wallaert, Behavioral Psychologist for Bing, fired back in defense of Bing It On. He noted that Bing later morphed its claim into, "Wherever we go, people prefer Bing over Google for the web's top searches," and points to several posts that document the changes.
Next, Wallaert addressed the small sample size of 1,000 people to obtain the results. "A 1,000 person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is just 1,500 people."
Ayres is irked that "Microsoft's ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000." But Wallaert also seems irked, claiming that there is no way Bing can post those numbers because it does not track the results. He wrote:
Sign up for CIO Asia eNewsletters.