An Introduction to Apache's Newest Project: Metron

Let’s start with a brief on the latest from the Apache umbrella – Apache Metron.

Per Apache Software Foundation, Apache Metron is a cyber security application framework that provides organizations the ability to ingest, process and store diverse security data feeds at scale in order to detect cyber anomalies and enable organizations to rapidly respond to them.

Hortonworks, the sponsor of Metron’s incubation, says Metron is built to address the Advanced Persistent Threats (APT) using machine learning.

You must be wondering as to what all is it leading to. Basically, certain activities of hackers go beyond those encountered conventionally, such as sustained cyber-attacks on specific organizations (such as large retailers, etc. that have significant online presence or are reasonably digitized) with malicious intent. On top of that, they are very difficult to detect through conventional means or mechanisms.  Regardless of the aim of such attacks, the damages and their degrees are varied. Some examples being collateral damage to brand (owing to loss of customer data), and denting investor confidence – wiping away millions in market-capitalization, besides of course the bottom line taking a severe hit.

Now that we know what these APTs can do to your business from a cyber-security point-of-view, let’s see how Metron helps in addressing these challenges to a great extent.

First and foremost, it addresses the most basic, and yet the most critical aspect of cyber-security: real-time monitoring of access and activities.

That is fine. But, the question arises – what does it specifically do to overcome the APTs?

It does quite a few things that are not possible with traditional systems:

• Get live data, process, and update in live dashboards using scalable architecture (storage and processing)
• Configuration driven (no need for additional coding)
• Supports time-series analysis (from 5 seconds to 100 years)
• Agile support that helps when requirements are dynamic, thus overlooked items can be caught such as frequent login failures of a particular user who was idle for quite long periods of time.

For the record, it is still in the Beta stage (Beta 2 in August 2016) with the first beta release (Apache Metron 0.1) in April 2016. Since it is still in the incubation stage (at the Apache Software Foundation), there are quite likely many more constraints in its effective implementations.

While an exhaustive list cannot be arrived at just yet, here’s what some early adopters (such as us) encountered:

• No community support – as mentioned above, owing to it being still in the Beta stage, one can expect that there is no support for the developer community as yet.
• Presently there are no supporting templates, which are expected to be included in due course 

Despite the fact that it hasn’t quite matured yet, there are the obvious bright spots such as:

• Development effort reduced (by as much as 85%)
• Easy to operate and implement (configuration-driven)
• Dynamic requirements supported
• Tight integration of the several parts of the solution such as Kafka, Storm, Kibana, etc., with in-built features to separate the data

Is it limited to just security analytics? Not at all. There are other areas or domain where the applicability of Metron can be explored. From what we have seen, it can be inferred that regardless of the industry vertical, Apache Metron will find extensive application in areas where there is live streaming of data as well as functions where continuous monitoring is warranted. Examples being transportation systems, ICU (health-care) monitoring, etc.

As the platform evolves, one can expect the both its effectiveness and employability to grow immensely. Exciting times ahead for advanced analytics (security, forecasts, predictions, and much more).