DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Power BI Embedded Analytics — Part 1: Introduction and Power BI Authoring Overview
  • Real-Time Analytics: All Data, Any Data, Any Scale, at Any Time
  • Real-Time Edge Application With Apache Pulsar
  • Spring Boot Application With Kafka, Elasticsearch, Redis With Enterprise Standards Part 1

Trending

  • Mastering Advanced Traffic Management in Multi-Cloud Kubernetes: Scaling With Multiple Istio Ingress Gateways
  • Ensuring Configuration Consistency Across Global Data Centers
  • Next-Gen IoT Performance Depends on Advanced Power Management ICs
  • Unlocking the Benefits of a Private API in AWS API Gateway
  1. DZone
  2. Data Engineering
  3. Data
  4. Visualizations on Apache Kafka Made Easy with KSQL

Visualizations on Apache Kafka Made Easy with KSQL

How to get started with processing streaming data with Confluent KSQL and visualizing it using the Arcadia Data platform.

By 
Shant Hovsepian user avatar
Shant Hovsepian
·
May. 23, 18 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
10.3K Views

Join the DZone community and get the full member experience.

Join For Free

Shant Hovsepian is the CTO and co-founder of Arcadia Data and is going to tell us about how to get started with processing streaming data with Confluent KSQL and visualizing it using the Arcadia Data platform.

The first Kafka Summit in London was just last month, and a popular topic at the show was KSQL. Released for production use with Confluent Platform 4.1, KSQL gives Kafka users a streaming SQL engine so they can use a SQL-like language to process and query data in Kafka. It's been getting a lot of traction since, and thanks to a lot of hard work and dedication, it was just released in production-ready form as part of Confluent Platform 4.1. This is a huge step in simplifying many types of stream processing that can be run on Kafka. With the wealth of existing expertise in SQL, developers can now use those skills with KSQL to more quickly build applications for data filtering, transformation, enrichment, manipulation, and analysis of Kafka data.

KSQL is a game-changer not only for application developers but also for non-technical business users. How? The SQL interface opens up access to Kafka data to analytics platforms based on SQL. Business analysts who are accustomed to non-coding, drag-and-drop interfaces can now apply their analytical skills to Kafka. So instead of continually building new analytics outputs due to evolving business requirements, IT teams can hand a comprehensive analytics interface directly to the business analysts. Analysts get a self-service environment where they can independently build dashboards and applications.

Arcadia Data is a Confluent partner that is leading the charge for integrating visual analytics and BI technology directly with KSQL. We've been working to combine our existing analytics stack with KSQL to provide a platform that requires no complicated new skills for your analysts to visualize streaming data. Just as they will create semantic layers, build dashboards, and deploy analytical applications on batch data, they can now do the same on streaming data. Real-time analytics and visualizations for business users have largely been a misnomer until now. For example, some architectures enabled visualizations for end users by staging Kafka data into a separate data store, which added latency. KSQL removes that latency to let business users see the most recent data directly in Kafka and react immediately.

We're hearing a lot of excitement about important business uses. For example, customers are looking to optimize operations by immediately analyzing the effects of their latest applications. With the feedback loop enabled by KSQL, they are able to adjust and optimize more quickly. Others are looking to detect operations errors such as discrepancies in transaction information in real time versus end-of-day problem resolution. In some cases, the goal isn't necessarily real-time access, but simply reducing time-to-analysis from hours to minutes.

If you want to experience this for yourself, getting started is easy. You can download binaries that get you exploring visualizations on Kafka very quickly. The binaries consist of the free desktop analytics tool, Arcadia Instant, plus Docker images that have containerized versions of Kafka and KSQL. This means you can set up a test environment in minutes that runs entirely on your desktop. You don't have to worry about setting up a cluster of nodes or about setting up cloud instances - you can just start experimenting. As long as you have about 8 GB RAM on your system, then you have plenty of power to run this setup. There are also a few walkthroughs that will guide you through the process of setting up a visualization environment.

Getting Started With Arcadia Instant in Combination With the Confluent Platform

To get started, please follow our Get Running with KSQL guide. It will tell you how to create a working setup of Arcadia Instant and the Confluent Platform components KSQL and Kafka (plus sample data).

From there, build an example dashboard by following our guide, A Day in the Life of a Business Analyst Using Streaming Analytics. You can also watch the video version of the guide. Even if you have no prior experience with Arcadia Data, you will find the guide to help you quickly get up to speed on building a real dashboard. More information is available on our streaming visualizations web page.

In summary, if you're an application developer who works with Kafka, or hopes to work with it soon, KSQL is the right technology choice for many types of processing you'll need to do. And if you want to expose your analytics capabilities to a non-technical audience, you no longer have to build your own visualization outputs in something like Excel or spend a lot of time bolting on a tool designed for other environments. The best choice is to go with a proven technology like Arcadia Data running with KSQL to provide that "last mile" of streaming data capabilities to your end users. So take a look at what Confluent and Arcadia Data have done, and stay tuned for more innovations in the coming months!

If you're interested in what KSQL can do, check out:

  • Download the Confluent Platform and follow the Quick Start to get started with KSQL
  • Watch the introductory video at the KSQL home page
  • Check out the clickstream demo that’s available as part of KSQL’s GitHub repo
  • Questions? Ask them in the #ksql channel on our community Slack group
kafka Visualization (graphics) Data (computing) application Analytics

Published at DZone with permission of Shant Hovsepian, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Power BI Embedded Analytics — Part 1: Introduction and Power BI Authoring Overview
  • Real-Time Analytics: All Data, Any Data, Any Scale, at Any Time
  • Real-Time Edge Application With Apache Pulsar
  • Spring Boot Application With Kafka, Elasticsearch, Redis With Enterprise Standards Part 1

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!