VANCOUVER, BC -- Last year's foundation of the Open Data Platform Initiative (ODPi), a collaborative project of The Linux Foundation that aims to reduce complexity surrounding the Hadoop ecosystem, made waves in certain parts of the Apache Software Foundation (ASF) concerned by the creation of an external organization that could exert influence over Apache projects.
At the Apache: Big Data North America conference in Vancouver, BC this week, the ODPi moved to ease those concerns through dialog and sponsorship of the ASF.
The idea behind ODPi's creation was to provide a big data kernel in the form of a tested reference core of Apache Hadoop, Apache Ambari and related Apache source artifacts. ODPi released a runtime specification and test suite earlier this year.
Big-name big data members
The organization has dozens of members. Most of them are big data solution providers like Hortonworks, Pivotal, EMC, IBM and SAS. While there are some end users also in the mix — and the organization is encouraging new members to join, including more end users — one of the concerns within ASF is that ODPi now employs the majority of committers to Hadoop ecosystem projects.
"Right now, ODPi is this sort of super organization of Hadoop vendors," Jim Jagielski, senior director in the Tech Fellows program at Capital One and one of the developers and founders of ASF, said in a panel about the issue yesterday. "Worst case, there could be a concerted effort by a single entity to basically create a Hadoop ecosystem that ODPi wants and not necessarily what the community wants."
"It's something that we will be looking at," he added. "The thing that really differentiates the ASF model from a lot of the other models out there is that it really is focused on the individual developers."
For instance, right now, there are two somewhat overlapping projects around authorization and data security in the Hadoop ecosystem: Apache Sentry, a top-level project supported by Hadoop distribution vendor Cloudera, and Apache Ranger (incubating), a project supported by Hadoop distribution vendor Hortonworks. Cloudera is not a member of ODPi but Hortonworks is. If ODPi certification becomes a differentiator for end users, the potential exists for ODPi to give favor to Apache Ranger implementations, regardless of merit.
That said, Jagielski also said he felt that scenario was unlikely.
"I think ODPi and the folks that are spending big money behind everything actually realize that they would be shooting the gift horse in the mouth," he said.
The nuclear option
It should also be noted that ASF holds the nuclear option of wiping the slate of committers to a project and establishing a whole new team.
Sign up for CIO Asia eNewsletters.