Security Information and Event Management (SIEM) with Elastic
Implementation of Security Information and Event Management with Elastic Stack
Join the DZone community and get the full member experience.Join For Free
With increased Cybersecurity challenges, firms are constantly battling to bring down the Mean Time to Detect/Discover (MTTD) of security threats. This is critical for multiple aspects such as customer satisfaction, legal compliance, and creditability of the organizations. The organization needs to identify, communicate and mitigate an issue before the user does.
Before getting into nuances of SIEM on Elastic, let us refresh the basics:
SIEM: SIEM comprises of Security Information Management (SIM) and Security Event Manager (SEM) to provide real-time security alerts generated by data, applications, and network infrastructure.
Security Information Management: Storage, analysis, and reporting of log data (inclusive of audit logs which are essentials to analyze the anomalies).
Security Event Manager: Analyzes the data collected through SIM, provides real-time monitoring and alerts based on business rules.
To have a better SIEM implementation, the following are some key steps:
The below capabilities might sound similar to feature engineering in Data Science since SIEM detection rules and engines are built on similar objectives.
- Log Management: Logs are the basic foundation for SIEM. As much as we want to keep all information about a transaction, let it be manual or automated, we have constraints on infrastructure and performance. Hence it is important to understand what logs we want to capture which will help us in the future to be able to detect an anomaly or threat. Also, this involves log collection, aggregation, storing and retaining, adhering to compliance, rotating, and reporting logs. There are key recommendations for effective log management inclusive of centralized log aggregation.
- Security Data Analytics (Correlation and Visualization through Alerts & Dashboards): This step is to translate the collected data into meaningful analysis by identifying and combining the factors. Correlation always doesn’t mean causation but this helps in identifying the threats/events and alerting them promptly. Power of analysis is based on how effectively we can report through real-time informational charts such as key patterns, correlations, and forecasting.
- Incident Detection, Response & Forensic Analysis: Performing a root cause analysis and generating an incident report that provides a detailed analysis of an attack attempt or an ongoing attack that helps enterprises take appropriate remedial action immediately. This helps to reduce the MTTD.
- Threat Intelligence: This isn’t part of traditional SIEM. However, while an efficient SIEM reduces the MTTD by analyzing internal information adding it with threat intelligence that focuses on external context helps in preventing the threats.
We explored Elastic Stack five years ago for one of our smart search solutions and I liked the performance and scalability apart from the intended functionalities of enterprise search, auto-completion/suggestion/correction. The capabilities such as schema-free, Restful APIs, and big data management are quite powerful.
Elastic Stack has evolved beyond search in the last couple of years. With added products for Elastic Observability and Elastic Security, Elastic has become quite powerful in the cybersecurity space.
Elastic Security focuses on prevention, detection, and response to threats through Elastic SIEM and endpoint security.
Elastic Stack & Security Information and Event Management:
Below is the high-level structure of components in Elastic SIEM
Log Management: Primarily done by Logstash which collects, parses, and transforms logs and by beats, the lightweight agents that acquire data and feed to Logstash and Elasticsearch. Beats are purpose-built data shippers for various kinds of data such as logs, metrics data, network data, windows event logs, audit data, uptime monitoring data, and cloud data. This helps to achieve the foundation of gathering data from systems and infrastructure which is the foundation for SIEM.
Incident Detection, Response & Forensic Analysis: Elastic endpoint provides visibility and advanced threat detection. Apart from the user-defined rules, the pre-built rules are added for better threat detection and response. Also proactively it blocks malware and ransomware.
Security Data Analytics (Correlation and Visualization through Alerts & Dashboards): Done combined by the endpoints, elastic search & Kibana.
Okay, it’s time to get some hands-on experience!
Elastic offers a trial of cloud deployment to explore and evaluate Elastic Security.
The deployment comes with Elastic Search, Kibana & Application Performance Monitoring by default.
The APM agents can be created and configured as below:
Apart from APM agents, I was looking for options to add other agents to be able to ingest, process, and detect anomalies (an end-to-end SIEM). Well, I was quite impressed with the available options. A quick snapshot as below:
Elastic is leveraging the beats (Filebeat for logs, MetricBeat for metrics, Winlogbeat for Windows Event Log, and so on)
For the trial version, we have an option of loading our data or explore the sample data provided by Elastic. I explored weblogs to understand the visualization, APM agents and endpoint configuration, and the power of ML anomaly detection.
The configuration and visualization were pretty straightforward with plenty of help & widgets available to support.
As stated earlier, though threat analytics isn’t part of SIEM by definition, it has become an unsaid expectation for SIEM solution to have anomaly detection as part of it.
Elastic lets us create ML jobs for prediction models. For the forecasted anomalies we can configure the rules so that the false positive is minimized.
- Of course, one size doesn’t fit all. However Elastic provides great capabilities for log management, security analytics & incident detection, and forecasting, an end-to-end SIEM solution.
- There was a blog written on Elastic Stack for SIEM in 2018: https://dzone.com/articles/using-the-elk-stack-for-siem two years ago. It’s quite interesting to see how far Elastic SIEM has evolved over a while. In fact, with a lot of firms moving towards solutions such as Google Cloud Platform (GCP), Azure & Zoom to support remote collaboration, Elastic Security 7.10 has come up with key capabilities to detect and prioritize the threats across solutions: https://www.elastic.co/blog/whats-new-elastic-security-7-10-0-correlation-cloud-visibility-detection
- The most critical element is to understand the domain, business problem, influencing factors, and building appropriate rules to identify threats. While we may have fantastic technology in hand, it depends on the foundation to ensure an effective SIEM solution.
- Elastic SIEM: https://www.elastic.co/guide/en/siem/guide/7.8/siem-overview.html
- Featured Image Courtesy: Unsplash photo by Tobias Tullius
Opinions expressed by DZone contributors are their own.