Data is being created as fast as it is being collected. High velocity and streaming data could become obsolete minutes after it was created as in the movement of markets on a stock trading floor, or multimedia streaming used for surveillance and security. The challenge with this is to be able to take action on the insights from information that is ever changing.
However, even the variety and velocity of data may be the least of an organisation's concerns. In a recent IDC study, which polled 300 organisations from all industries across Australia, 47 per cent of respondents revealed they do not have the skill sets required to manage big data.
While the DBA was quite adept and trusted to manage the structured data in a relational database, he is suddenly out of his depth when it comes to mapping and contextualising large volumes of data of different types and sources.
The dilemma is that the skill sets required to manage big data are not those a DBA can typically up-skill for, which leaves many organisations exposed when it comes to dealing with the new unstructured data coming into their environment.
The skill sets required to manage big data are less typical of our preconceived image of what an IT professional should be. While there is still an element of technical acumen needed, the actual tasks involved in measuring and interpreting unstructured data are more suited to people skilled in mathematics and physics. It is more scientific than technical, giving rise to such titles as data scientist, data modeller or analytician.
In an attempt to accelerate the availability of these skill sets in the market, many vendors are incubating universities with programs and courses to attract students to big data as a lucrative career option, but it is yet to be seen how quickly supply will meet demand in a landscape that is becoming more competitive.
Furthermore, people who have these skill sets or come out of universities where these skills have been honed to perfection will be resources in high demand. We are more likely to see these people make a name for themselves through new start-ups in the style of industry luminaries such as Gates, Jobs, Ellison and Page.
This raises a very interesting scenario for end-user organisations. If skilled graduates are absorbed by the vendor community or go into business for themselves, this leaves a significant skills gap for end-user organisations to fill; and until they are able to do so, they face the very real possibility of costly consulting engagements in their big data projects. End-user organisations will be challenged to circumvent this scenario with strategies to attract skilled professionals.
Sign up for CIO Asia eNewsletters.