Other Hadoop-related projects at Apache include:
- Avro: A data serialization system.
- Cassandra: A scalable multi-master database with no single points of failure.
- Chukwa: A data-collection system for managing large distributed systems.
- HBase: A scalable, distributed database that supports structured data storage for large tables.
- Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.
- Mahout: A scalable machine-learning and data-mining library.
- Pig: A high-level data-flow language and execution framework for parallel computation.
- ZooKeeper: A high-performance coordination service for distributed applications.
Do Your Homework
As with any technology, Hadoop adopters need to be aware of vulnerabilities in the tool itself, as well as the myriad compatibility and configuration problems that can crop up with any such tool.
"Like some of the GRC [governance, risk and compliance] installs we've seen, this can bomb enormously and be a massive money waste," says Alex Hutton, Zions' director of technology and operations risk and governance.
His advice? Do your homework before rushing in. Take all the necessary time to flesh out a detailed road map for the data you're looking to process, carefully review how Hadoop will behave with the rest of your network, and develop a clear taxonomy model and strict metrics for it to follow.
Hutton says Zions achieves that by using a combination of custom controls and the vocabulary for event recording and incident sharing (VERIS) framework, which provides a common language for describing security incidents in a structured and repeatable manner.
"Custom controls and VERIS are our ontologies for metrics. FAIR [factor analysis of information risk] is our risk ontology. Specific metrics support the conceptual categories these ontologies describe," he says.
If you don't have these things, Hutton adds, you are not ready to use big data as a security tool.
Because Wood's team did all its homework before rolling out the new warehouse, Zions enjoyed a relatively smooth deployment. As long as other companies also do their homework, they can hope for similar success, Hutton says.
Sign up for CIO Asia eNewsletters.