DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Comparison of Alternatives for Stream Processing and Streaming Analytics

Comparison of Alternatives for Stream Processing and Streaming Analytics

Kai Wähner user avatar by
Kai Wähner
CORE ·
Sep. 11, 14 · Interview
Like (4)
Save
Tweet
Share
24.00K Views

Join the DZone community and get the full member experience.

Join For Free

The demand for stream processing is increasing a lot these days. Frameworks (Apache Storm, Spark) and products (e.g. IBM InfoSphere Streams, TIBCO StreamBase, Software AG Apama) for stream processing and streaming analytics are getting a lot of attention these days. The reason is that often processing big volumes of data is not enough. Data has to be processed fast, so that a firm can react to changing business conditions in real time. This is required for trading, fraud detection, system monitoring, and many other examples. A “too late architecture” cannot realize these use cases.

There is not much literature available about stream processing and streaming analytics in 2014. Analyst Forrester has published a report recently: The Forrester Wave™: Big Data Streaming Analytics Platforms, Q3 2014. Leaders are Software AG (Apama), IBM (InfoSphere Streams), TIBCO (StreamBase), SAP (Event Stream Processor) and Informatica.

Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse

I have written an article for InfoQ, which gives some more details about stream processing and streaming analytics:

Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse

This article discusses what stream processing is, how it fits into a big data architecture with Hadoop and a data warehouse (DWH), when stream processing makes sense, and what technologies and products you can choose from. The following is just a snippet of the article describing the components of a stream processing solution.

Comparison of Alternatives for Stream Processing / Streaming Analytics

Stream processing can be implemented by doing-it-yourself, using a framework or a product. Doing-it-yourself should not be an option in most cases, because there are good open source frameworks available for free. However, a stream processing product might solve many of your issues out-of-the-box, while a framework still requires a lot of self-coding and the Total Cost of Ownership might be much higher than expected compared to a product.

From a technical perspective, the following components are required to solve all "streaming challenges" and implement a stream processing use case:

  • Server: An ultra-low-latency application server optimized for processing real-time streaming event data at high throughputs and low latency (usually in-memory).
  • IDE: A development environment, which ideally offers visual development, debugging and testing of stream processing processes using streaming operators for filtering, aggregation, correlation, time windows, transformation, etc. Extendibility, e.g. integration of libraries or building custom operators and connectors, is also important.
  • Connectors: Pre-built data connectivity to communicate with data sources such as database (e.g. MySQL, Oracle, IBM DB2), DWH (e.g. HP Vertica), market data (e.g. Bloomberg, FIX, Reuters), statistics (e.g. R, MATLAB, TERR) or technology (e.g. JMS, Hadoop, Java, .NET).
  • Streaming Analytics: A user interface, which allows monitoring, management and real-time analytics for live streaming data. Automated alerts and human reactions should also be possible.
  • Live Data Mart and/or Operational Business Intelligence: Aggregates streaming data for ad-hoc, end-user, query access, alerting, dynamic aggregation, and user management. Live stream visualization, graphing, charting, slice and dice are also important.

As of end-2014, only a few products are available on the market that offer these components. Often, a lot of custom coding is required instead of using a full product for stream processing.

Apache Storm, Spark, IBM InfoSphere Streams, TIBCO StreamBase, SAG Apama

The two most-widespread open source frameworks for stream processing are Apache Storm and Spark. IBM InfoSphere Streams, TIBCO StreamBase and Software AG's Apama are important players from proprietary vendors.

Please read my InfoQ article for more details about these products, and how they relate to a Data Warehouse (DWH) and Apache Hadoop. Here is the link: Real-Time Stream Processing as Game Changer in a Big Data World with Hadoop and Data Warehouse.

As always, I appreciate all feedback and discussions...



Stream processing Big data Analytics Comparison (grammar)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Public Cloud-to-Cloud Repatriation Trend
  • Spring Boot Docker Best Practices
  • Using QuestDB to Collect Infrastructure Metrics
  • Core Machine Learning Metrics

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: