SQL and Hadoop Query Performance Smackdown
See which SQL engine won this smackdown where MapReduce, Spark, LLAP, Tez, and Presto competed to see who performed fastest with SQL queries!
by
Join the DZone community and get the full member experience.
Join For FreeLLAP Wins the Fastest Execution Among the SQL Engines!
Comcast is one of the nation’s leading providers of communications, entertainment, and cable products and services. Headquartered in Philadelphia, PA, they employ over 100,000 employees nationwide whose goal is to deliver the highest level of service and improve the customer experience. Comcast decided to run what they call their “Hadoop Query Performance Smackdown” for SQL engines.
The Comcast Big Data team uses an enterprise data like with over 1000+ daily active users running on 70 racks with petabytes of usable enterprise data available via Hive tables. Their uses cases range from ad hoc jobs, batch and streaming data, and reporting. They wanted to pick a SQL engine which would give them the best performance for the most practical use cases. They ran tests against MapReduce, Spark, LLAP, Tez, and Presto. The end result was to pick a SQL engine to recommend to the CTO and the community.
They used a test methodology which utilized TPC-DS queries defined in the Hive benchmark for each of the SQL engines. Each query was run one at a time to utilize all the resources from the cluster. The team ensured that care was taken to tune and configure the engines. Furthermore, each query was run three times to make sure there were no anomalies. By doing this, the team calculated an average run time from the three rounds. Take notice of how the tests that were run against LLAP have much faster execution times that the other engines. LLAP had been explained as only been best optimized for ORC. The Comcast team found that they achieved much better performance across the board. LLAP was by far the fastest engine against Tez and Presto. SparkSQL did not manage to complete the benchmark successfully.
To learn more, read the Datanami article.
In addition, watch the Comcast youtube video session from the Hortonworks Dataworks Summit on June 14, 2017, in San Jose, CA to learn about how you can use these results to help guide your company’s big data initiative on that journey of supporting interactive queries.
Published at DZone with permission of Roni Fontaine, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments