Few other technologies generate the same level of fear, he said. "It's very different to say, 'Look, we are a community of responsible scientists who are building safety into this thing, and we're pretty sure it's going to work,'" Atkinson said.
The good news is that humans are still in control over how AI and robots will develop, but a more robust discussion about AI's future is needed, said Stuart Russell, a professor of electrical engineering and computer science at the University of California, Berkeley.
Even though Atkinson suggested that the danger is limited because it's still impossible to design a robot with intentionality, Russell suggested intentions aren't necessary for there to be a risk.
"If the system is better than you at taking into account more information and looking further ahead into the future, and it doesn't have exactly the same goals as you, then you have a problem," Russell said. "The difficulty is that we don't know what the human race's values are, so we don't know how to specify the right goals for a machine so that its behavior is something that we actually like."
In some cases, AI developers might think they're giving the right instructions to an intelligent machine, but the results aren't what they expected, like in the legend of King Midas, Russell said. "What happens when you don't like what they're doing?" he said. "You could say, 'Shut them down,' but a super intelligent system ... knows that one of the biggest risks to it is being shut down, so it's already outthought you."
With many AI researchers working on a small piece of the general-purpose intelligence puzzle, policymakers and scientists should talk about the potential negative implications instead of "keeping our fingers crossed that we'll run out of gas before we run off the cliff," Russell added.
Some people are more optimistic about super intelligent machines coexisting with humans, said Manuela Veloso, a computer science professor at Carnegie Mellon University. Service robots now escort visitors at Carnegie Mellon to Veloso's office and surf the Web to learn new information, she noted.
Robots are reaching a point where they will provide benefits to many people, she said. Research on coexistence will teach intelligent machines "not be taught to be outside of the scope of humankind but to be part of humankind," she said. "We will have humans, dogs, cats and robots."
Sign up for CIO Asia eNewsletters.