SQL and Hadoop Query Performance Smackdown
SQL and Hadoop Query Performance Smackdown
See which SQL engine won this smackdown where MapReduce, Spark, LLAP, Tez, and Presto competed to see who performed fastest with SQL queries!
Join the DZone community and get the full member experience.
Join For FreeLearn how error monitoring with Sentry closes the gap between the product team and your customers. With Sentry, you can focus on what you do best: building and scaling software that makes your users’ lives better.
LLAP Wins the Fastest Execution Among the SQL Engines!
Comcast is one of the nation’s leading providers of communications, entertainment, and cable products and services. Headquartered in Philadelphia, PA, they employ over 100,000 employees nationwide whose goal is to deliver the highest level of service and improve the customer experience. Comcast decided to run what they call their “Hadoop Query Performance Smackdown” for SQL engines.
The Comcast Big Data team uses an enterprise data like with over 1000+ daily active users running on 70 racks with petabytes of usable enterprise data available via Hive tables. Their uses cases range from ad hoc jobs, batch and streaming data, and reporting. They wanted to pick a SQL engine which would give them the best performance for the most practical use cases. They ran tests against MapReduce, Spark, LLAP, Tez, and Presto. The end result was to pick a SQL engine to recommend to the CTO and the community.
They used a test methodology which utilized TPC-DS queries defined in the Hive benchmark for each of the SQL engines. Each query was run one at a time to utilize all the resources from the cluster. The team ensured that care was taken to tune and configure the engines. Furthermore, each query was run three times to make sure there were no anomalies. By doing this, the team calculated an average run time from the three rounds. Take notice of how the tests that were run against LLAP have much faster execution times that the other engines. LLAP had been explained as only been best optimized for ORC. The Comcast team found that they achieved much better performance across the board. LLAP was by far the fastest engine against Tez and Presto. SparkSQL did not manage to complete the benchmark successfully.
To learn more, read the Datanami article.
In addition, watch the Comcast youtube video session from the Hortonworks Dataworks Summit on June 14, 2017, in San Jose, CA to learn about how you can use these results to help guide your company’s big data initiative on that journey of supporting interactive queries.
What’s the best way to boost the efficiency of your product team and ship with confidence? Check out this ebook to learn how Sentry's real-time error monitoring helps developers stay in their workflow to fix bugs before the user even knows there’s a problem.
Like This Article? Read More From DZone
Published at DZone with permission of Roni Fontaine , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}