Over a million developers have joined DZone.

At-Least-Once Enhancement in Concord Streaming

Concord has added the very popular at-least-once guarantees for messaging with Apache Kafka.

· Big Data Zone

Read this eGuide to discover the fundamental differences between iPaaS and dPaaS and how the innovative approach of dPaaS gets to the heart of today’s most pressing integration problems, brought to you in partnership with Liaison.

Concord.io has upgraded their framework to support at-least-once guarantees for messaging, a highly critical feature for many users.  They are using the widespread tool Apache Kafka.

Image title

You don't need to change any code to use the new swappable distributed runtime for this new capability, it's just a different flag. The Concord runtime isolates constraints, key-value storage, messaging, monitoring, and all the system's architecture.  The developer then just writes their code isolated in all the modern enterprise languages: Java, Scala, Clojure, C++, Ruby, Python, and Go. 

Image title

Concord removes a lot of the complexities you normally find in complex streaming. In Concord, as opposed to say Akka, you don't have to worry about clustering, supervision, deployment, and other operations tasks that are difficult in distributed environments. You just configure what Zookeeper is pointing to in your Kafka cluster. You can use Apache Kafka as a standalone cluster or use the Kafka clustering in HDP. Developers don't directly use Kafka, as it is used by the Concord nodes and is abstracted for users to remove complexity.

Image title

In the new upgraded Concord, you have the choice of at-most-once and at-least-once systems. The main difference underneath the covers is that the at-most-once uses point-to-point communication between Concord nodes. Kafka is very solid and will allow additional brokers to be added for more capacity. You can also scale Concord dynamically at runtime using the Concord command line shell.

To get started with Concord, simply fork their GitHub and get running locally!

I will be talking to them again on running a simple example on Concord with Kafka and the same sample in Spark Streaming (maybe Flink, Storm, or others), then have them present at the Future of Data Meetup in Princeton with a follow up article on DZone. Currently only Concord supports writing streaming applications in C++, Java, Scala, Clojure, Python, Go, and Ruby; as Spark Streaming supports Java, Python, and Scala. What I like about Concord is that you get many of the features of a PaaS with fast streaming, pluggable runtimes and multiple container support. I would love to see this run on YARN.

Image title

Discover the unprecedented possibilities and challenges, created by today’s fast paced data climate and why your current integration solution is not enough, brought to you in partnership with Liaison

streaming,streaming analytics,java,big data

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}