Now, seven years later, we can see that Cloudera's founders were right: We have found that Hadoop and the movement it started has a valuable role in mainstream enterprises.
We are in a revolution on several fronts. Traditional enterprise RDBMS software now has competition: open source, big data software. Much to my surprise and pleasure, the hacker and enterprise software traditions are no longer distinct, but have merged. No longer is there a strict division between those who develop software and those who use it. We regularly see Cloudera customers collaborating with our engineers. Users are often directly involved in software advances.
No single software component dominates. Hadoop is perhaps the oldest and most successful component, but new, improved technologies arrive every year. New execution engines like Apache Spark and new storage systems like Apache Kudu (incubating) demonstrate that this software ecosystem evolves rapidly, with no central point of control. Users get better software sooner.
The new software is not only more affordable and scalable, it offers a better style. Institutions can explore messy, diverse data sources, perform experiments, and rapidly develop and evolve applications. Data from sensors, social media, and production can be combined to develop insights, inform decisions, and fuel new products. Companies like Cloudera have helped this software meet the requirements of industry, making it more stable, reliable, manageable, secure, and easily integrated with existing systems.
Government and industry themselves are transforming. Not only are new companies like Uber and Tesla using data to reinvent their sectors, but established companies like Caterpillar and Chevron are dramatically improving themselves through data technology. Much of the progress we will make in this century will come from increased understanding of the data we generate.
Looking back, 10 years ago I never would have guessed that Hadoop would form a critical part of such huge trends. I am incredibly surprised and proud of how far we have come. I look forward to following Hadoop's continued impact as the data century unfolds.
Sign up for CIO Asia eNewsletters.