XiaoIce is different from the Siris of the world because it's more of a friend than a personal assistant. It can hold conversations, tell jokes, suggest products to buy and do other things. The New York Times even reported that about 25% of users have at some point told XiaoIce "I love you."
Unlike Google's research project, which gleans responses from movie dialog, XiaoIce gets them from social media in China. So when you ask XiaoIce. "What's the meaning of life?" the A.I. scans a database of people who have posed that question online, and chooses one of the popular responses to provide to the user.
The disturbing reality is that XiaoIce is not only basing its replies on social media chatter, it's replacing social media and messaging for some users in some circumstances. And therein lies the dystopian risk with messaging-based chatbots.
Why messaging apps are both a great and horrible place for A.I. chatbots
I think it's clear that the future of chatbots will combine both personal assistance and conversational functions. Chatbots will replace both search engines and social networking.
Messaging apps are a great place to place an A.I. chatbot because we already spend so much time in those apps. We've accepted the bare-bones, text-only UI, which eliminates some of the uncertainties of virtual assistants that rely on voice recognition and spoken replies.
But messaging chatbots also come with risks. Because human beings are complex creatures plagued by cognitive biases, irrational thinking and emotional needs, the line between messaging with a friend and messaging with A.I. will be fine to nonexistent for some people.
Chatbot users will find gratification in their XiaoIce-like chatbots for the same reason people love dogs. Chatbots will make people feel like they're interacting with another person, a real friend. But unlike real humans, who can be self-centered and detached, chatbots will probably have dog-like loyalty and selflessness. They will always be there for you and will always have time for you. The combination of intelligence, loyalty and faithfulness is irresistible to the human mind.
So the risk with messaging chatbots is that they could facilitate a preference for maintaining a relationship with A.I., rather than with other humans, at least for some people.
Now that Google is getting ready to release a messaging-based chatbot into the world, the idea of messaging-based chatbots will move from the fringes into the mainstream. And that will be the beginning of our long, complicated and fraught relationship with machines that talk to us.
Sign up for CIO Asia eNewsletters.