Tom White, a principal program manager for Microsoft Bing has been employing the input of the crowd for several years. The company had been using its own "private crowd" of judges to rate search result relevance to significant success.
The key for Microsoft is that crowd tasks are well-defined and highly targeted. "We developed a very focused expertise with our judges so they could work on one task and one task only," says White.
What White needed was the option to throw any number of things to the crowd--some new feature or user interface a developer or designer was experimenting with--at a few thousand folks to find out what worked and what didn't. "It had to do with scale and flexibility," he explains. "We wanted a larger pool of people we could use for a wide variety of tasks."
Getting Agile With the Crowd
Using the internal platform White's team had already developed for Bing's private crowd, a user interface designer with an idea for a better user experience will plug an "a versus b test" in and send it off to the crowd for evaluation.
Bing also employs the crowd to supplement its machine learning processes with some real, live human feedback. The company may take a large set of potential search queries to classify (with the crowd charged to determine, for example, if a query is "adult-oriented") and use that to inform automated classification.
It's the kind of work Bing couldn't simply outsource or offshore because of its high volume, says White. "The offshore approach doesn't scale as effectively as the crowdsourcing approach. We can access thousands of people in markets it may be difficult for offshore providers to reach," he says. The crowd is also cheaper and faster than using internal employees, delivering the kind of quick input that developers and designers need.
Besides, White says, using a crowd makes more sense because it's more reflective of his company's user base. "The size of the crowd gives us cognitive diversity," White says. "It's like we have thousands of people in a room versus dozens."
Lionbridge, for example, says it has 100,000 crowd workers who have been screened and qualified to complete tasks for clients. And crowdsourcing providers have a strong incentive to source skilled workers, says Martha Crow, Lionbridge's senior vice president of global enterprise solutions, because customers like Bing pay based on actual output rather than headcount or by the hour.
"We break down the work into a task, we pay our workers based on quality tasks completed, and we get paid by our clients in the same way," Crow says. "This model completely eliminates the nonproductive time and puts the burden of throughput and quality work on the workers and the service provider."
Sign up for CIO Asia eNewsletters.