DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
What's in store for DevOps in 2023? Hear from the experts in our "DZone 2023 Preview: DevOps Edition" on Fri, Jan 27!
Save your seat
  1. DZone
  2. Data Engineering
  3. Databases
  4. Apache Calcite Report: Part 1

Apache Calcite Report: Part 1

Tim Spann interviews Julian Hyde and Josh Elser, two Apache Calcite leads, and reports on why they believe Apache Calcite and Avatica are important.

Tim Spann user avatar by
Tim Spann
CORE ·
Jul. 03, 16 · Interview
Like (5)
Save
Tweet
Share
6.58K Views

Join the DZone community and get the full member experience.

Join For Free

I spoke with Julian Hyde (Slides) and Josh Elser (Slides), both are Apache Calcite PMCs.

We spoke about Apache Calcite and Avatica. Avatica is a sub-project of Calcite for historical reasons but is beginning to be used standalone, e.g. for high-performance connectivity to Phoenix/HBase. Avatica provides a wire API between clients and a server. The Avatica server is an HTTP server that accepts API calls in two popular formats of JSON or Protocol Buffers (Protobuf).  The Avatica client is a standard JDBC driver. Clients can be developed in many languages since protobuf and JSON over HTTP are pretty common. The open source community has already added a Go Avatica client focused on Apache Phoenix.  There is also a very early .NET driver.

Image title

Talk with Julian Hyde

Julian has spoken about StreamingSQL (with Apache Calcite) and Query Optimization on Phoenix at some big conferences including XLDB, Hadoop Summit Dublin and the Kafka Summit.  He is the Apache Calcite Team lead and an Architect at Hortonworks.


Why Does Calcite Matter?

  • It provides a query planner for any database engine.   

  • It provides the heavy lifting for many of the key database features you come to expect.

  • It provides a SQL Query Parser and Abstract Syntax Tree (AST)

  • Calcite implements JDBC/ODBC frameworks

  • Data in multiple formats, multiple engines

  • It supports relational, document and KV stores

  • It supporst a variety of workloads

  • You can easily introduce new data formats and engines

  • Deconstructioned traditional database engine

  • You have visibility into all the components of the database engine

  • MetaData Catalog, authorization, algorithms (distributed join), scheduler / resource allocation, Engine, data format, storage

  • Key value is to separate queries from how to get data.

  • Your application shouldn't be tied to lower-level database implementations

Talk with Josh Elser

Josh has worked on Accumulo and Hbase and is now working on Phoenix Query Server Internals, along with Apache Calcite and Avatica.

The primary value of this work is to allow non-JVM developers to interact with Hadoop and other big data servers using familiar Enterprise tools and standards like ODBC and BI Tools.

You can query all your data sources at the same time.

Phoenix QueryServer is built on Apache Calcite's Avatica sub-project. It is a generic Jetty web server with wire API to allow client talk to server and interacts with databases via a JDBC Driver.

Anyone can implement an HTTP Clients to call it in all the popular langues. This will allow .NET developers, Python and Ruby scripters to access HBase.

Why are there so few drivers or they are not great. Every new database has to build their own stack from scratch. The leading open source databases Postgresql and MariaDB have good ODBC drivers, but those took years and a lot of developer hours. Avatica provides a reference implementation JDBC Driver Protocol Buffers are recommended, they are backwards compatible and are easier to parse than JSON. Avatica provides a metrics systems to provide utilization and information on SQL queries that are running. This should become the Universal Client for database access. The hard work is defining the a documented, clear, stable wire protocol. So drivers can be implemented in a few weeks, not months or even years.

References

  • quidem: scripting language for testing databases

  • Apache Calcite on Github

  • Apache Calcite Project

  • Apache Calcite Introduction and Demo

  • Planning with Polyalgebra

  • Streaming SQL

  • Streaming SQL with Apache Calcite

  • Apache Calcite:  One Size Fits All Planner

  • Introduction to Calcite

  • Apache Phoenix Query Server
  • Apache Calcite:  One Query Planner

Apache Calcite Relational database

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How to Submit a Post to DZone
  • What Was the Question Again, ChatGPT?
  • 7 Awesome Libraries for Java Unit and Integration Testing
  • Deploying Java Serverless Functions as AWS Lambda

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: