Big Data is a relative term. We have been managing data growth since back in the 1880s when American statistician, Herman Hollerith, developed a mechanical tabulator based on punched cards to tabulate statistics from millions of pieces of data.
Hollerith's tabulator was used in the United States Census in 1890, resulting in it being completed months ahead of schedule and significantly under budget. Hollerith went on to become the founder of the company that would later become IBM.
From those early days of data collection and analysis, there has been an increasing appetite by organisations and individuals to collect and analyse data for a variety of purposes such as operational efficiency, sales and marketing and forecasting.
As organisations found new uses for data, and began to ask more complex questions of it, technology also evolved to meet the demand in being able to manage it, and this led to a symbiotic relationship between information and technology.
Data thus far had been a measurable quantum and was organised in a structured manner that was easily identifiable and retrievable. Technology evolved at a commensurate pace to ensure the equation was balanced and stayed relatively the same.
However, this relied on all relevant data to be managed centrally, where it could be neatly indexed, filed in rows and columns in multiple layers, and then called upon and manipulated to give any number of results based on the questions asked.
This is the typical modus operandi of a relational database management system (RDBMS), which will store an organisation's operational data to be used to derive insights at various stages of the information lifecycle.
Regardless of where, when and how information was collected, it was rendered to the database structure to comply with standard query language. This standard of information management gave rise to the database administrator (DBA), who became somewhat a demigod in the IT department as he managed the loads and bottlenecks in the flow of information to an organisation's mission-critical applications.
This also brought about bespoke technologies that automated many of the DBA's functions and allowed him to concentrate on more important tasks in managing an ever growing data warehouse.
As queries became more complex and multi-dimensional, business intelligence tools came to the forefront to give organisations insights on the data they collected and stored to help manage their business operations through meaningful insights.
Thus far the DBA existed in relative comfort, orchestrating the flow of information in and out of the data warehouse with the help of many management tools that kept the organisation running at an optimum level. Then something unexpected happened. To understand this, we need to look quickly at what IDC classifies as the third platform of technology.
Sign up for CIO Asia eNewsletters.