Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Facebook releases query engine to open source community

Thor Olavsrud | Nov. 7, 2013
Facebook has released Presto -- a distributed SQL query engine optimized for running ad-hoc interactive queries on data sources up to petabytes in size -- to the open source community under the Apache license. It says Presto is 10x better than Hive.

Inside Apache Hive

Apache Hive, also initially developed at Facebook, has become the most common infrastructure for querying massive data warehouses on Hadoop clusters. But Presto's execution model is fundamentally different than Hive, Traverso says, making it better-suited to interactive queries.

"Hive translates queries into multiple stages of MapReduce tasks that execute one after another," Traverso says. "Each task reads inputs from disk and writes intermediate output back to disk. In contrast, the Presto engine does not use MapReduce. It employs a custom query and execution engine with operators designed to support SQL semantics. In addition to improving scheduling, all processing is in memory and pipelined across the network between stages. This avoids unnecessary I/O and associated latency overhead. The pipelined execution model runs multiple stages at once and streams data from one stage to the next as it becomes available. This significantly reduces end-to-end latency for many types of queries."

Facebook began working on Presto in fall 2012 and had a production system up and running early in 2013. By spring of this year, it had rolled Presto out to the entire company-more than 1,000 employees use it daily, running more than 30,000 queries processing one petabyte daily. Facebook has deployed the engine in multiple geographical regions and notes that it has successfully scaled a single cluster to 1,000 nodes.

The engine still has its challenges. Traverso says the main restrictions at this stage are a size limitation on join tables and cardinality of unique keys/groups. Additionally, it lacks the ability to write out data back to tables. But those issues are all front-and-center in Facebook's roadmap for Presto. Over the next few months, Facebook plans to remove restrictions on join and aggregation sizes and add the ability to write output tables.

"We are also working on a query "accelerator" by designing a new data format that is optimized for query processing and avoids unnecessary transformations,"Traverso says. "This feature will allow hot subsets of data to be cached from backend data store and the system will transparently use cached data to "accelerate" queries. We are also working on a high-performance HBase connector."

 

Previous Page  1  2 

Sign up for CIO Asia eNewsletters.