DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • What Are SpeedUp and ScaleUp in DBMS?
  • Advanced Maintenance of a Multi-Database Citus Cluster With Flyway
  • Non-blocking Database Migrations
  • Rails Asynchronous Processing

Trending

  • The Man Who Invented Selenium Also Built Its Replacement
  • Columnar Formats for AI Telemetry: Parquet vs Vortex With DuckDB
  • A Reference Architecture for Constrained AI-Assisted Design
  • How to Keep AI Agents From Going Rogue Using Docker Sandbox

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. Databases
  4. Running Streaming ETL Pipelines with Apache Flink on Zeppelin Notebooks

Running Streaming ETL Pipelines with Apache Flink on Zeppelin Notebooks

A step-by-step tutorial for running Streaming ETL with Flink on Zeppelin. Let's dive deeper into the Flink interpreter in Zeppelin Notebooks.

By 
Jeff Zhang
·
Jul. 09, 20 · Tutorial
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
7.1K Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Apache Zeppelin 0.9 comes with a redesigned interpreter for Apache Flink that allows developers and data engineers to use Flink directly on Zeppelin notebooks for interactive data analysis. Over the next paragraphs, we describe why Streaming ETL is a great fit for stream processing frameworks like Apache Flink and we dive deeper into the Flink interpreter in Zeppelin Notebooks by showcasing a tutorial of how developers can run Streaming ETL data pipelines with Flink on Zeppelin.

zeppelin image


Streaming ETL and Apache Flink

Extract-transform-load (ETL) is a common operation related to massaging and moving data between storage systems. ETL jobs have historically been triggered periodically, frequently copying data from transactional database systems to an analytical database or a data warehouse.

Streaming ETL pipelines serve a similar purpose traditional ETL: they transform and enrich data and can move it from one storage system to another. However, streaming ETL pipelines are different from traditional ETL in that they operate continuously and are capable of both reading records from sources that continuously produce data as well as moving the data, with low latency, to their desired destination.

Streaming ETL is a common use case for Apache Flink because of its ability to address most common data transformation or enrichment tasks with Flink SQL (or Table API) and its support for user-defined functions. Additionally, Flink provides a rich set of connectors to various storage systems such as Kafka, Kinesis, Elasticsearch, and JDBC database systems. It also features continuous sources for file systems that monitor directories and sinks and write files in a time-bucketed fashion. Let us now describe how the Flink interpreter works in Zeppelin notebooks. 


The Flink Interpreter in Zeppelin 0.9

The Flink interpreter can be accessed and configured from Zeppelin’s interpreter settings page. The interpreter has been refactored so that Flink users can now take advantage of Zeppelin to write Flink applications in three languages, namely Scala, Python (PyFlink), and SQL (for both batch & streaming executions). Zeppelin 0.9 now comes with the Flink interpreter group, consisting of the below five interpreters:

Flink Interpreter Class, Zeppelin Notebooks, Apache Zeppelin, Apache Flink


The Flink interpreter has been brought to a new level and now provides support for both writing Flink applications in three languages (Scala, Python [PyFlink] and SQL) as well as running such applications in different execution modes, such as running Flink in Local, remote or Yarn Mode. if you are interested in learning more about the Flink interpreter and for more information on how to get started with Zeppelin and all the execution modes for Flink applications you can check this article.


Flink on Zeppelin for Stream ETL pipelines

The Flink interpreter allows developers to build stream processing Flink applications, such as streaming ETL and real-time data analytics directly on Zeppelin Notebooks. This can be easily executed by taking advantage of Flink SQL along with specific UDFs (user-defined functions). Let us now showcase how you can execute streaming ETL using Flink on Zeppelin: 

You can use Flink SQL to perform streaming ETL by following the steps below (for the full tutorial, please refer to the Flink Tutorial/Streaming ETL tutorial of the Zeppelin website):

  1. Create a source table to represent the source data.
    We will use %flink.ssql to represent the following SQL are streaming SQL which will be executed via StreamTableEnvironment. You can write multiple SQL statements in one paragraph, just separate them with a semicolon.
    Create Kafka Source Table, Kafka Sources, Flink, Apache Flink, Zeppelin Notebooks

  2. Create a sink table to represent the processed data.
    Kafka Sink, Kafka Sink Table, Flink, Apache Flink, Apache Zeppelin

  3. After creating the source and sink table, we can use insert them to our statement to trigger the streaming processing job as the following: 


  4. After initiating the streaming job, you can use another SQL statement to query the sink table to verify your streaming job. 
    Here you can see the top 10 records which will be refreshed every 3 seconds.

This article discusses how the redesigned Flink interpreter works in Zeppelin 0.9.0 and provides some step-by-step guidance on how to perform streaming ETL jobs with Flink and Zeppelin. You can find additional tutorials, such using Flink on Zeppelin for batch processing, as well as performing more advanced operations with Flink on Zeppelin (e.g. resource isolation, job concurrency & parallelism, multiple Hadoop & Hive environments and more) on this series of posts. We hope you enjoy this new addition to the Zeppelin project and you try the Flink interpreter soon! 

Extract, transform, load Apache Flink Database Pipeline (software) Data (computing) Stream processing sql application career

Opinions expressed by DZone contributors are their own.

Related

  • What Are SpeedUp and ScaleUp in DBMS?
  • Advanced Maintenance of a Multi-Database Citus Cluster With Flyway
  • Non-blocking Database Migrations
  • Rails Asynchronous Processing

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook