DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > At-Least-Once Enhancement in Concord Streaming

At-Least-Once Enhancement in Concord Streaming

Concord has added the very popular at-least-once guarantees for messaging with Apache Kafka.

Tim Spann user avatar by
Tim Spann
CORE ·
Jul. 14, 16 · Big Data Zone · News
Like (5)
Save
Tweet
3.57K Views

Join the DZone community and get the full member experience.

Join For Free

Concord.io has upgraded their framework to support at-least-once guarantees for messaging, a highly critical feature for many users.  They are using the widespread tool Apache Kafka.

Image title

You don't need to change any code to use the new swappable distributed runtime for this new capability, it's just a different flag. The Concord runtime isolates constraints, key-value storage, messaging, monitoring, and all the system's architecture.  The developer then just writes their code isolated in all the modern enterprise languages: Java, Scala, Clojure, C++, Ruby, Python, and Go. 

Image title

Concord removes a lot of the complexities you normally find in complex streaming. In Concord, as opposed to say Akka, you don't have to worry about clustering, supervision, deployment, and other operations tasks that are difficult in distributed environments. You just configure what Zookeeper is pointing to in your Kafka cluster. You can use Apache Kafka as a standalone cluster or use the Kafka clustering in HDP. Developers don't directly use Kafka, as it is used by the Concord nodes and is abstracted for users to remove complexity.

Image title

In the new upgraded Concord, you have the choice of at-most-once and at-least-once systems. The main difference underneath the covers is that the at-most-once uses point-to-point communication between Concord nodes. Kafka is very solid and will allow additional brokers to be added for more capacity. You can also scale Concord dynamically at runtime using the Concord command line shell.

To get started with Concord, simply fork their GitHub and get running locally!

I will be talking to them again on running a simple example on Concord with Kafka and the same sample in Spark Streaming (maybe Flink, Storm, or others), then have them present at the Future of Data Meetup in Princeton with a follow up article on DZone. Currently only Concord supports writing streaming applications in C++, Java, Scala, Clojure, Python, Go, and Ruby; as Spark Streaming supports Java, Python, and Scala. What I like about Concord is that you get many of the features of a PaaS with fast streaming, pluggable runtimes and multiple container support. I would love to see this run on YARN.

Image title

kafka

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Composable Architecture
  • Is Java Still Relevant?
  • How to Submit a Post to DZone
  • Password Authentication: How to Correctly Do It

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo