DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • OpenTelemetry Moves Past the Three Pillars
  • Maintaining ML Model Accuracy With Automated Drift Detection
  • Observability Agent Architecture
  • Tobiko Data: Revolutionizing Data Transformation With SQLMesh

Trending

  • AI, ML, and Data Science: Shaping the Future of Automation
  • Measuring the Impact of AI on Software Engineering Productivity
  • Java's Quiet Revolution: Thriving in the Serverless Kubernetes Era
  • Microsoft Azure Synapse Analytics: Scaling Hurdles and Limitations
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Monitoring and Observability
  4. A Hands-On Guide to OpenTelemetry: Exploring Telemetry Data With Jaeger

A Hands-On Guide to OpenTelemetry: Exploring Telemetry Data With Jaeger

This article continues the journey by learning how to effectively use visualized instrumentation telemetry data in Jaeger on your observability journey.

By 
Eric D.  Schabell user avatar
Eric D. Schabell
DZone Core CORE ·
Paige Cruz user avatar
Paige Cruz
·
Aug. 23, 24 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
5.2K Views

Join the DZone community and get the full member experience.

Join For Free

Are you ready to start your journey on the road to collecting telemetry data from your applications? Great observability begins with great instrumentation! 

In this series, you'll explore how to adopt OpenTelemetry (OTel) and how to instrument an application to collect tracing telemetry. You'll learn how to leverage out-of-the-box automatic instrumentation tools and understand when it's necessary to explore more advanced manual instrumentation for your applications. By the end of this series, you'll have an understanding of how telemetry travels from your applications to the OpenTelemetry Collector, and be ready to bring OpenTelemetry to your future projects. Everything discussed here is supported by a hands-on, self-paced workshop authored by Paige Cruz. 

The previous article programmatically instrumented and configured an application to direct all telemetry data to a Jaeger instance for visual insights. In this article we'll zoom into the usage of Jaeger to help developers when visually exploring their telemetry data.

It is assumed that you followed the previous articles in setting up both OpenTelemetry and the example Python application project, but if not, go back and see the previous articles as it's not covered here.

Exploring Jaeger

While we installed Jaeger in the previous article, we did not dive into its background or details. Jaeger is an open-source distributed tracing system and graduated CNCF project. It supports multiple telemetry storage backends and provides a web-based front end. It natively supports receiving OTLP formatted telemetry as we are using it in our examples with in-memory storage (default).

Starting from the point we left off in the previous article, we had our Python application, hello-otel, running, which our developer had programmed in our trace instrumentation. We had set up our trace exporter and used the OTLPSpanExporter to push our telemetry data in an OpenTelemetry Protocol (OTLP) format. 

Next, in our pod configuration, we added a Jaeger all-in-one instance that is designed for use for testing in development environments. It includes a collector, query, and UI components that we spun up to explore. From that point, we opened a browser and viewed the Jaeger UI at http://localhost:16686, which displayed the Gopher Detective as shown below:

Jaeger Gopher DetectiveWe then generate telemetry data (tracing spans) by accessing our application and make several requests to the http://localhost:8001/doggo endpoint. 

Searching Through Traces

Back in our Jaeger UI, we can select hello-otel from the service dropdown menu and note that this Traces landing page provides us with a search panel. This is where you can filter to look at traces with a specific service, a particular operation, relevant tags (e.g., status http.status_code=500), duration, or any combination of the above:

Traces landing page

The traces matching our search query above populate a scatter plot and a table. The scatter plot is a quick way to visually identify traces that look out of the ordinary, and by clicking on the bubble, you'll be taken straight to the corresponding trace waterfall. The table view is helpful for sorting to find specific traces by duration, amount of spans, or recency.

Traces can feature hundreds of services and thousands of spans, so the default view is to have all span details collapsed providing an overview of the request results with quick stats about the trace like duration and number of services. From this view you can see what spans are taking up the most time relative to the overall request duration:

View showing what spans are taking up the most time relative to the overall request duration

The search bar here allows you to search within the trace for spans with properties like GET, 200, etc. Clicking the target opens up the matching spans as shown:

Matching spans

Let's look closer at the visual options for exploring our traces.

Scatter Plots and Graphs

The scatter plot is a quick way to visually identify traces that look out of the ordinary and by clicking on the bubble, you'll be taken straight to the corresponding trace waterfall. The table view is helpful for sorting to find specific traces by duration, amount of spans, or recency:

Table view

Clicking on a trace name takes you to its trace waterfall page, and selecting two or more checkboxes lets you compare the selected traces with each other:

View showing selected traces to compare

Once we've zeroed in on an interesting span, clicking the name opens up more details like the associated tags and process information. This is where your manually instrumented metadata becomes a powerful way to inspect your system behavior:

/doggo tags

Special Visualizations

Now, you might have noticed there is a drop-down next to the trace search bar. Clicking on this reveals a specific list of visualizations that are available, including a few we will detail below.

  • Trace graph
  • Trace spans table
  • Trace flamegraph

The trace graph view shows the trace with spans groups into node blocks with options to color the nodes uniquely by service, by time to highlight the critical path, or self time which shows the longest span durations not waiting on children:

Trace graph

The trace spans table shows a table with duration, operation, and service name per span in the trace. The option to search by service or operation lets you zoom in on specific interactions. Clicking a span ID takes you to the trace detail view with that span highlighted:

Trace spans table

A traces flamegraph view is another way to visualize the trace waterfall. As you explore the spans, you can right-click to collapse unnecessary details, copy the function name to use in another query, or highlight similar spans within the trace:

Traces flamegraph

Now let's try a more advanced use case where we compare several traces for change over time.

Advanced Searches

When new bugs stick their head up or unexpected behavior happens in your systems, your investigation typically is trying to answer questions like:

  • What changed?
  • How does this compare to normal?

This is where trace comparison shines. Use this to compare two or more traces to quickly identify which spans are present or occur more frequently in only one of the traces. Below we are starting to ask ourselves, why did one request to our /doggo application take 685ms and another only 281ms?

2 traces selected for comparison

By selecting the two traces and clicking on compare traces, we can see a visual graph representation with color coding modeled after code diffs: 

  • Grey - Nodes in both Trace A and Trace B
  • Red - Nodes only in Trace A
  • Green - For nodes only in Trace B

Visual graph representation with color coding modeled after code diffs

As we have discovered, trace data powers both high-level insights into relationships between services and low-level insights. It would be overwhelming to show all of that information in one view and it's the ability to jump between trace comparisons, span queries, individual attributes on method calls, and topology maps that make trace data flexible and powerful.

These examples use code from a Python application that you can explore in the provided hands-on workshop. 

What's Next?

This article detailed a lot of the usage of Jaeger to help developers when visually exploring their telemetry data. In our next article, we will dive into manually instrumenting metadata in our tracing spans for our application.

Open source Telemetry Data (computing) Instrumentation (computer programming)

Published at DZone with permission of Eric D. Schabell, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • OpenTelemetry Moves Past the Three Pillars
  • Maintaining ML Model Accuracy With Automated Drift Detection
  • Observability Agent Architecture
  • Tobiko Data: Revolutionizing Data Transformation With SQLMesh

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!