DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. The Challenge of Log Management in Modern IT Environments

The Challenge of Log Management in Modern IT Environments

Check out this post to learn more about common challenges in log management.

Daniel Berman user avatar by
Daniel Berman
·
Feb. 06, 19 · Analysis
Like (1)
Save
Tweet
Share
5.45K Views

Join the DZone community and get the full member experience.

Join For Free

Gaining visibility into modern IT environments is a challenge that an increasing number of organizations are finding difficult to overcome.  

Yes — the advent of cloud computing and virtually “unlimited” storage has made it much easier to solve some of the traditional challenges involved in gaining visibility. However, architecture has evolved into microservices, containers, and scheduling infrastructure. Software stacks and the hardware supporting them are becoming more complex, creating additional and different challenges. These changes have directly impacted the complexity of logging and it takes a very specific set of tools and strategies to be able to solve this visibility challenge.

Understanding the Challenge

Log data is one of the cornerstone requirements for overcoming this challenge. Yet nearly every application, appliance, and tool today generates an ever-increasing stream of log messages, containing a wealth of information on what happened, when, and why. These sources are often distributed on-premise, on the cloud, or across different clouds.

In today’s world, old school methodologies are no longer viable:

  • Distributed systems, whether based on the traditional server/client model, containers, and/or cloud services, generate a huge amount of logs that are not only expensive to store but also query.
  • Effective monitoring must be done in real time. If an application crashes, teams need to be able to be effectively alerted so as to perform a fast troubleshooting process.
  • Logs are generated by devices, applications, servers, and in different formats. If logs are simply shipped as flat files to sit on a file server, it takes significant effort to perform any kind of analysis.
  • Most importantly: cloud-based architecture requires efficient logging, alerts, automation, analysis tools, proactive monitoring, and reporting. Old school logging supports none of this.

Centralized Logging Concepts

Modern log management must include log analysis capabilities and perform aggregation, processing, storage, and analysis. These components must be designed on top of cloud principles: high availability (HA), scalability, resiliency, and automation.

  • Aggregation – the ability to collect and ship logs from multiple data sources.
  • Processing – the ability to transform log messages into meaningful data for easier analysis.
  • Storage – the ability to store data for extended time periods to allow for monitoring, trend analysis, and security use cases.
  • Analysis – the ability to dissect the data by querying it and creating visualizations and dashboards on top of it.
  • Alerting – the ability to get notified when an event is taking place in real-time  

Cloud Principles for a Log Management Solution

Now that we’ve covered the core elements comprising a modern log management system, we will cover the cloud principles that must be considered when trying to build a visibility solution into your environment. This will illustrate the challenges they bring to maintaining the solution.

  • High availability – building for High Availability (HA) means eliminating single points of failure and that failures cause no, or little to no, disruption of service. In cloud environments such as AWS, this means building a log management solution that is deployed on more than one availability zone and possibly more than one region and includes a data replication mechanism using a service such as S3.
  • Scalability – This is critical for most cloud services and the hardest to manage in log management solutions. This is because some services are either bound by storage, indexing, and/or clustering, which requires careful management behind the scenes. Other services, such as forwarders and log shippers, are easily scaled to meet demands.
  • Upgrades – Upgrades are common activities in nearly every service. When you are talking several interconnected visibility components with clustering, storage, reindexing, etc., the effort becomes a large project in itself. 
  • Resilience – This principle refers not only to the storage layer (uptime and loss prevention) but also to avoiding service disruption. Where this differs from HA, it is the ability for the architecture of your solution to continue to operate in the face of updates, migrations, zone failures, or the ability to recover data in the case of deletion/corruption.

Summing it Up

It should be apparent that managing a full-scale solution for log management in modern software requires a considerable amount of planning. It doesn’t mean it can’t be done — of course, it can be done. It just means you need to plan carefully and make a knowledgeable decision.

For helping you decide whether to implement your own do-it-yourself log analytics solution or opt for a SaaS provider such as Logz.io, here is a checklist to help you get the most out of your operations.

Your log management solution must:

  • Support cloud-native log aggregation for containers, container orchestration, and cloud services.
  • Support a processing engine, which is able to transform logs as necessary (ETL or grokking)
  • Support storage that is cloud-native or auto-replicated between nodes. Ideally, storage can be scaled automatically.
  • Support data retention that is configurable, preferably archivable utilizing cheaper storage tiers.
  • Support advanced analytics such as anomaly detection and machine learning.
  • Support integration into notification tools, such as Slack or Hipchat (sometimes called ChatOps), email.
  • Support custom metrics and dashboards and include pre-built dashboards and integrations with cloud services.
IT Cloud computing microservice Data (computing)

Published at DZone with permission of Daniel Berman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Handling Automatic ID Generation in PostgreSQL With Node.js and Sequelize
  • Image Classification With DCNNs
  • Secrets Management
  • What Is Policy-as-Code? An Introduction to Open Policy Agent

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: