Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Big data gets runtime specification

Thor Olavsrud | March 29, 2016
The ODPi Runtime Specification and test suite is designed to help organizations write Hadoop-based applications once with the confidence that they will run on a variety of Hadoop distributions.

The Open Data Platform Initiative (ODPi) released its first ODPi Runtime Specification and test suite Monday as part of its goal to ensure a standard deployment model for enterprise big data applications across Apache Hadoop distributions.

"This is the culmination of this whole year's work," says John Mertic, senior manager of ODPi.

The nonprofit ODPi formed last year in an effort to reduce the amount of complexity surrounding the Hadoop and big data environment. The idea was to provide a big data kernel in the form of a tested reference core of Apache Hadoop, Apache Ambari and related Apache source artifacts.

The kernel, called ODPi Core, would be used to simplify upstream and downstream qualification efforts - a "test once, use everywhere" core platform that could eliminate the growing fragmentation in the space. Applications and tools built on the reference platform should integrate with and run on any compliant system.

In September of last year, ODPi officially became a collaborative project of the Linux Foundation.

Mertic explains ODPi is an effort to bring together constituents from all the various "party lines" with a stake in the big data ecosystem.

"What we really wanted to do was to make sure we could have the community well represented," he says. "The biggest feedback that we got was that each distro does things slightly differently; they name their files differently; their APIs behave differently."

The new runtime specification descends from Apache Hadoop 2.7 and features HDFS, YARN and MapReduce components. Mertic says the test framework and self-certification aligns closely with the Apache Software Foundation by leveraging Apache Bigtop for comprehensive packaging, testing and configuration. More than half the code in the latest Bigtop release originated in ODPi. The ODPi Runtime-Compliance tests are linked directly to lines in the ODPi Runtime Specification. To assist with compliance, ODPi has also provided a reference build.

The organization says the published specification includes rules and guidelines on how to incorporate additional, non-breaking features, which are allowed provided source code is made available through relevant Apache community processes.

"It was a little over a year ago that ODPi was formed, and we have already proved beneficial to upstream ASF projects (Hadoop, Bigtop, Ambari)," says Roman Shaposhnik, director of Open Source at Pivotal, and an Apache Hadoop and Bigtop committer and ASF member. "This is why the first release of the ODPi Runtime Specification and test suite is so exciting. It is a big step toward realizing our goal of accelerating the delivery of business outcomes through big data solutions by driving interoperability on an enterprise-ready core platform."

 

1  2  Next Page 

Sign up for CIO Asia eNewsletters.