Moving Streaming Analytics Out of the Data Center

this blog focuses on moving streaming analytics outside the confines of the traditional data center. moving streaming analytics closer to where data originates can be accomplished by leveraging an enterprise-grade data movement application, married with an extremely lightweight streaming engine. this combination is being used by forward-looking organizations to solve usage cases in a number of areas, including:

• cyber security – identify a malicious intrusion before or as it occurs.
• fraud – analyze streaming transactions to determine which needs immediate attention.
• predictive maintenance – predict outlier conditions from streaming machine and sensor data.
• customer experience and marketing – use streaming data insights to personalize interactions.
• stream data management – transform and clean data in motion, storing only what you need.

ioat (internet of anything) data includes any new data source generated from sensors and machines, server logs, clickstream web application servers, social media, as well as files and email. by reacting rapidly, a high degree of analytic intelligence such as ioat data can be collected from its origin to help companies gain fast insights to outpace their competition. whether it’s personalizing a “next best offer” or providing an immediate treatment to an early warning alert, ioat data is only beginning to impact bottom line revenues. the time to act is now, but what components do we need to add to our modern data applications?

hortonworks data flow

hortonworks data flow (hdf), powered by apache nifi, is a data application platform designed to solve data acquisition and delivery challenges, inside and outside the data center. it provides a fast, easy and secure way to move data from anywhere it originates to anywhere it needs to go. hdf has a simple gui command and control for building “data workflows” so there is no need to write custom data movement scripts. hdf provides simple event processing (sep) out of the box to curate data payloads as they move from one place to another.

figure 1 : act fast on streaming data with hdf and sas esp

sas event stream processing

sas event stream processing (esp) analyzes and understands streaming data as it is being generated, detects patterns of interest as they occur, and provides the necessary instructions to take the correct actions; i.e. what alerts to issue and what portions of data should be retained for further investigation.

hdf and sas esp

by incorporating sas esp into our edge originating hdf based data workflows, we are able to inject complex event processing (cep) into any ioat application. it’s important to note that both hdf and sas esp are ideal candidates to run on edge node gateways, which typically have very small memory footprints and run lightweight linux kernels. since hdf and sas esp are architected to run from the smallest of devices large clustered configurations, they are a perfect compliment both inside and outside the data center.

summary

in summary, hdf, powered by apache nifi, is a secure, reliable enterprise data movement platform, which provides simple event processing (sep) processors out of the box. hdf is architected to run from the data center to the edge of an ioat framework and back. sas esp is an extremely fast streaming analytics engine, also architected to run in the data center and out to an edge node device and back. sas esp and hdf are seamlessly integrated via sas esp nifi processors. together, they can provide immediate, streaming, deep actionable intelligence for improved customer experience and lower cost operations.