The role of data scientists will be transformed as machine learning techniques become more widely used by businesses, according to Stephen Brobst, CTO of analytics firm Teradata.
While many of the principles behind 'AI' approaches are not new, interest within an enterprise setting has exploded in recent years. And as usage becomes more widespread and sophisticated, the role of data scientists will begin to evolve too, according to Brobst.
He explains that data scientists have typically spent much of their time 'wrangling' data to feed into predictive models. In future, more of this work will be automated and data scientists will instead be more focused on selecting which machine learning or deep learning tools to utilise for specific tasks.
"Instead of the data scientist spending most of their time working with the data itself, they are going to spend most of their time working with the algorithms - so you have to be much more sophisticated in algorithm selection and topology selection in a neural network and so on," he says.
"You still have to understand how the nature of the data influences your algorithm selection, but you are going to be spending less time preparing data, because data scientists today spend more than 60 percent of their time preparing data, beating data over the head, torturing it, shoving it into a hole, and that is not going to be required anymore."
He says that this is already becoming the case with advanced algorithms used for machine learning purposes currently, but as deep learning - a branch of machine learning - become more widely used, data scientists' priorities will shift further.
"In deep learning there will be fewer requirements for domain knowledge and more requirements for algorithm selection based on the type of data that you have and so on," he says, "so this shift of skill set will be very, very interesting."
Data science skills gap
For many enterprises, however, the challenge is just finding the right data science expertise in the first place. Accessing skills remains a major challenge, despite the efforts of tech companies to provide training courses and resources.
"There is a big [skills] gap right now, because the number of data scientists who are qualified versus the demand for them is way out of whack - and this is particularly true in the UK," Brobst explains. "Partially that is because the UK is relatively aggressive in understanding that data science is important, so [UK organisations] are kind of on the leading part of that."
It is not just a problem faced by the UK. "We are not producing enough [data scientists] worldwide," he says. "There is a worldwide hunt for a good data scientist going on."
Sign up for CIO Asia eNewsletters.