DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
The Latest "Software Integration: The Intersection of APIs, Microservices, and Cloud-Based Systems" Trend Report
Get the report
  1. DZone
  2. Data Engineering
  3. Data
  4. Application Log Analysis and Data Visualization

Application Log Analysis and Data Visualization

Log data is a useful source of info that provides insight into how customers interact with products. It also helps to troubleshoot the issues with the app and getting to the root of the problem.

Aman Sardana user avatar by
Aman Sardana
·
Oct. 10, 17 · Opinion
Like (2)
Save
Tweet
Share
8.29K Views

Join the DZone community and get the full member experience.

Join For Free

Application log data is a useful source of information that provides insight on how the customer interacts with the product. It also helps in troubleshooting the issues with the application and getting to the root cause of the problem.

Most enterprises rely on APM (Application Performance Management) tools to monitor the system health ensuring application availability and performance levels are met. While APM tools come with out of the box features that help in reporting important metrics like response times, service call volumes, errors etc., they don't offer any insight into the business errors or any other custom metrics in the context of the business. This is where Log Analysis comes to the rescue, providing better insight into the context of the business and answering the complex business questions.

There are many great tools in the market like Splunk that help in aggregating the logs and data from multiple servers and applications. Once the data is collected and aggregated, it can be searched, correlated, and visualized to find any operational issues impacting the business. While Splunk is by far the most widely used Log Management tool in most enterprises, there are many other good options available like Loggly, Logstash, ElasticSearch, and Kibana. These tools require the logs to be formatted before they are ingested into the tool for further analysis. There are scenarios where either the application is not integrated with the central log management tool or the manual analysis of logs to visualize the data in the context of the business provides a quick and deeper insight into the application issues.

We'll see a few examples of how the log data can be analyzed using a data visualization tool like Tableau to understand the application behavior and troubleshoot business errors.

Figure 1 shows the heat map for the number of times an error occurred every hour based on the logs data collected over several days.

This log data visualization reveals several interesting facts:

  • System Error is observed mostly during the busy hours of the day and not during the early morning hours.
  • When the bug resulting in the System Error was fixed, no errors were observed in the month of November.
  • On Oct 1 around 9:00 AM there are a large number of timeout errors in the system that occurred within an hour. On further investigation (Figure 2), it turns out that all these errors occurred for only 5 mins and not the full hour as seen in Figure 1. This suggests that something went wrong in the system resulting in a large number of errors in a short time interval.

LogAnalysis1Figure 1

LogAnalysis2Figure 2

Figure 3 shows the response time for the service calls grouped in bins on the x-axis and the number of calls on the y-axis. This view helps in analyzing the service calls with response time falling outside the acceptable SLA.

LogAnalysis3Figure 3

Figure 4 shows the visualization for the total number of service calls vs the errors seen in the logs over a period of time. This shows a correlation between the number of service calls and the total count of the errors on a specific day.

LogAnalysis4Figure 4

Figure 5 shows the number of service calls that timed out over a 24 hour period. The x-axis is the hour of the day and the y-axis is the minute of the hour. This visualization shows the system performance every minute of the hour spread throughout the day in a single view. The heat map helps in visualizing the error density and the correlation with the other errors.

LogAnalysis5Figure 5

Figure 6 shows the heat map for the number of events processed each minute of the hour. This visualization helps in understanding the event processing pattern throughout the day.

LogAnalysis6Figure 6

In Summary, APM and log analysis tools are good at monitoring system performance and the analysis of the well-formatted logs fed to the log management tool. There are scenarios where it is easy and helpful to visualize the raw data from the logs in tools like Tableau to get better insight into the operational metrics and discover unknown patterns from the data.

Data visualization application Log analysis

Published at DZone with permission of Aman Sardana. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Key Elements of Site Reliability Engineering (SRE)
  • Integrating AWS Secrets Manager With Spring Boot
  • How Agile Architecture Spikes Are Used in Shift-Left BDD
  • A First Look at Neon

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: