DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > Concord.io: Processing Data in Motion

Concord.io: Processing Data in Motion

Tim Spann interviews Concord co-founder Shinji Kim about their distributed stream processing framework and lists some quick specs about the technology.

Tim Spann user avatar by
Tim Spann
CORE ·
Jul. 01, 16 · Big Data Zone · Opinion
Like (6)
Save
Tweet
4.57K Views

Join the DZone community and get the full member experience.

Join For Free

Concord.io is very fast, is written in C++, and has an interesting new streaming framework that scales. There are a lot of stream processing frameworks, but this looks to be the only one written in fast C++ with APIs for multiple languages like Go and Java.

Image title

For code snippets, check out the walkthrough.

Image title

I spoke with Shinji Kim, Co-Founder of Concord Systems in NYC on what the fast deal is going on with Concord. Concord currently runs on Apache Mesos, but could run on other environments like YARN. The main use is for processing data in motion as it sits between message queues and databases. It's used for fast data enrichment, aggregation, filtering, and deduplication. Since it is written in C++ there is no garbage collection cycle to slow down processing and no worries about JVM heap memory management. It supports popular fast data sources like Kafka and Kinesis and data sinks like HDFS and MySQL. It uses the Pub/Sub Operator Model (composable jobs by metadata). Users can manage local state on a Key-Value store backed by RocksDB.

  • It supports writing applications in Python, Ruby, Go, Java, Scala and C++ using it's API. 

  • Containerized execution environment.

  • Uses Open Source Twitter Zipkin for monitoring.

  • Concord systems reports benchmarks of Concord (500K QPS/node at 10ms/event) vs Spark Streaming (100K QPS/node at 1s batch window).

The API is only missing the ubiquituous hipster NodeJS and R. If this product becomes open source, those APIs are very likely to appear.

It's an interesting product to consider if you are hitting some performance walls in streaming.

Data (computing) Stream processing

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How to Design a CRUD Web Service for Inheritable Entity
  • Instancio: Test Data Generator for Java (Part 2)
  • After COVID, Developers Really Are the New Kingmakers
  • Creating a REST Web Service With Java and Spring (Part 1)

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo