Observability Pillars: Exploring Logs, Metrics and Traces

The ability to measure the internal states of a system by examining its outputs is called Observability. A system becomes 'observable' when it is possible to estimate the current state using only information from outputs, namely sensor data. You can use the data from Observability to identify and troubleshoot problems, optimize performance, and improve security.

In the next few sections, we'll take a closer look at the three pillars of Observability: Metrics, Logs, and Traces.

What Is the Difference Between Observability and Monitoring?

‘Observability wouldn’t be possible without monitoring.’ 

Monitoring is another term that closely relates to observability. The major difference between Monitoring and Observability is that the latter refers to the ability to gain insights into the internal workings of a system, while the former refers to the act of collecting data on system performance and behavior.

In addition to that, Monitoring doesn't really think about the end goal. It focuses on predefined metrics and thresholds to detect deviations from expected behavior. Observability aims to provide a deep understanding of system behavior, allowing exploration and discovery of unexpected issues.

In terms of perspective and mindset, Monitoring adopts a "top-down" approach with predefined alerts based on known criteria. Observability takes a "bottom-up" approach, encouraging open-ended exploration and adaptability to changing requirements.

Monitoring detects anomalies and alerts you to potential problems. However, Observability detects issues and helps you understand their root causes and underlying dynamics.

Three Pillars of Observability 

Observability, built on the Three Pillars (Metrics, Logs, Traces), revolves around the core concept of "Events." Events are the fundamental units of monitoring and telemetry, each time-stamped and quantifiable. What distinguishes events is their context, especially in user interactions. For example, when a user clicks "Pay Now" on an eCommerce site, this action is an event expected within seconds.

In monitoring tools, "Significant Events" are key. They trigger:

• Automated Alerts: Notifying SREs or operations teams.
• Diagnostic Tools: Enabling root-cause analysis.

Imagine a server's disk nearing 99% capacity; it's significant, but understanding which applications and users cause this is vital for effective action.

1. Metrics

Metrics serve as numeric indicators, offering insights into a system's health. While some metrics like CPU, memory, and disk usage are obvious system health indicators, numerous other critical metrics can uncover underlying issues. For instance, a gradual increase in OS handles can lead to a system slowdown, eventually necessitating a reboot for accessibility. Similar valuable metrics exist throughout the various layers of the modern IT infrastructure.

Careful consideration is crucial when determining which metrics to continuously collect and how to analyze them effectively. This is where domain expertise plays a pivotal role. While most monitoring tools can detect evident issues, the best ones go further by providing insights into detecting and alerting complex problems. It's also essential to identify the subset of metrics that serve as proactive indicators of impending system problems. For instance, an OS handle leak rarely occurs abruptly. 

Tracking the gradual increase in the number of handles in use over time makes it possible to predict when the system might become unresponsive, allowing for proactive intervention.

2. Logs  

Logs frequently contain intricate details about how an application processes requests. Unusual occurrences, such as exceptions, within these logs can signal potential issues within the application. It's a vital aspect of any observability solution to monitor these errors and exceptions in logs. Parsing logs can also reveal valuable insights into the application's performance.

Logs often hold insights that may remain elusive when using APIs (Application Programming Interfaces) or querying application databases. Many Independent Software Vendors (ISVs) don't offer alternative methods to access the data available in logs. Therefore, an effective observability solution should enable log analysis and facilitate the capture of log data and its correlation with metric and trace data.

3. Traces

Tracing is a relatively recent development, especially suited to the complex nature of contemporary applications.  It works by collecting information from different parts of the application and putting it together to show how a request moves through the system.

The primary advantage of tracing lies in its ability to deconstruct end-to-end latency and attribute it to specific tiers or components. While it can't tell you exactly why there's a problem, it's great for figuring out where to look.

Integrating tracing used to be difficult, but with service meshes, it's now effortless. Service meshes handle tracing and stats collection at the proxy level, providing seamless observability across the entire mesh without requiring extra instrumentation from applications within it.

Each above discussed component has its pros and cons, even though one might want to use them all.

Observability Tools

Observability tools gather and analyze data related to user experience, infrastructure, and network telemetry to proactively address potential issues, preventing any negative impact on critical business key performance indicators (KPIs). 

Some popular observability tooling options include:

• Prometheus: A leading open-source monitoring and alerting toolkit known for its scalability and support for multi-dimensional data collection.
• Grafana: A visualization and dashboarding platform often used with Prometheus, providing rich insights into system performance.
• Jaeger: An open-source distributed tracing system for monitoring and troubleshooting microservices-based architectures.
• Elasticsearch: A search and analytics engine that, when paired with Kibana and Beats, forms the ELK Stack for log management and analysis.
• Honeycomb: An event-driven observability tool that offers real-time insights into application behavior and performance.
• Datadog: A cloud-based observability platform that integrates logs, metrics, and traces, providing end-to-end visibility.
• New Relic: Offers application performance monitoring (APM) and infrastructure monitoring solutions to track and optimize application performance.
• Sysdig: Focused on container monitoring and security, Sysdig provides deep visibility into containerized applications.
• Zipkin: An open-source distributed tracing system for monitoring request flows and identifying latency bottlenecks.

Conclusion

Logs, metrics, and traces are essential Observability pillars that work together to provide a complete view of distributed systems. Incorporating them strategically, such as placing counters and logs at entry and exit points and using traces at decision junctures, enables effective debugging. Correlating these signals enhances our ability to navigate metrics, inspect request flows, and troubleshoot complex issues in distributed systems.