Back in the early days, when forward-thinking executives pioneered the use of predictive analytics for business gain, they tended to recruit specialists who knew a lot about maths and economics but not necessarily very much about the business.
Today, of course, the art of predictive analytics is mainstream. Better still, it is starting to become readily available to anyone in the organisation, as needed.
But it has been a slow evolution. It began with line managers using spreadsheets to track activity after the event, and then - and no more scientifically than gauging wind direction by hand - trying to guess what might happen next.
We then had gifted amateurs trying to help by toying with simple databases and IT being goaded by line managers to at least 'look into' analytics.
Slowly, dashboards, disparate data silos and incompatible software tools accumulated but insights were still hard to uncover. Management staff knew more about their business but were no smarter.
Then board members saw the light. Directors were persuaded that data was one of their most valuable assets and that analysing it could lead to business gain.
The evolution gathered pace. Organisations made significant investments in predictive analytics and those people with oversight of these initiatives - whether in marketing, IT or elsewhere in the company structure - started finding themselves at the c-level, reporting directly to the board - and rightly so.
But if this is where we are today, there is still something missing.
Firstly and most visibly, there are simply not enough data scientists to meet employers' needs. Opinions about the scale of the shortage vary but a report by McKinsey & Company estimates that next year in the US there will be a shortage of 140,000 to 190,000 people with deep analytical skills - as well as 1.5 million managers and analysts with the knowhow to use the analysis of big data to make effective decisions.
Relatively, the situation is no different in Australia and New Zealand. Our teaching institutions, together with support from industry and government, are working to close this gap but as data volumes increase and ever more organisations seek to take advantage of them, the problem will persist. Supply is nowhere near catching up with demand.
In addition to the skills gap, the other things holding organisations and productivity back are variety and complexity - the need to keep pace with rapid technology advances and changing market conditions.
While useful insights were found in only customer and other internal data just a few years back, organisations now need to include data from many other sources and of many different types if they want optimum answers. Add to this the new need to include machine learning in modeling, and even well-resourced analytics infrastructures are stretched.
Sign up for CIO Asia eNewsletters.