OpenTelemetry Python: All You Need to Know About Tracing
How to get started quickly with OpenTelemetry and Python
Join the DZone community and get the full member experience.Join For Free
If you crack open OpenTelemetry, you’ll quickly discover that there’s a lot there. But, as a developer applying OpenTelemetry to your application, 99% of what’s in there doesn’t matter.
All you need to know is:
- Initialization: How to start and shutdown cleanly.
- Tracer methods: get_tracer, get_current_span, startSpan, and withSpan.
- Span methods: setAttribute, addEvent, recordException, setStatus, and end.
Seriously, that’s it. If you want to try it out, follow the guide below. A heavily commented version of the finished tutorial can be found at https://github.com/tedsuo/otel-python-basics, please use it as a reference when you get started with instrumenting your own application.
Off to Docker
(If you already have a python setup you’re fine with, just skip this bit).
This time, let’s do our local development in docker. Managing Python installations can be a bit of a snake’s nest, especially on a mac, where the current default python3 installation has a bit of an issue with psutil, which we depend on.
First, make a directory for your application:
Install docker, then grab a python image and start a container that mounts your application directly. If you’re in your app directory, the following starts a python container with your current directory mounted at /app, and logs you into a bash shell within the container.
Whenever you need to log into a new terminal, find the container ID and then use it to exec into a bash shell.
And that’s “all you need to know” about docker. :)
Once again, it is time to say Hello to this cruel World.
First, exec into your docker container and install the required dependencies. For this basic app, we’re going to use flask for the server, and requests for the client.
Create a file named server.py and make the world’s simplest app.
Amazing. Add a client at client.py which makes five requests in a loop:
In one docker terminal, start the server:
In another, run the client:
Ok so if you know python that was all very boring. Here’s the stuff you came for: installing OpenTelemetry.
First, you need to pick the analysis tool you want to target. I work on Lightstep, and we have free community accounts specifically for trying out OpenTelemetry like this. These instructions assume you have one of those. If you’d like to set up Jaeger instead, you can find installation instructions here.
To connect to lightstep, install the Lightstep distro for OpenTelemetry, the OpenTelemetry launcher. Lightstep is OpenTelemetry native, all the launcher does is install the relevant packages and make the configuration simpler.
The launcher will install the core OpenTelemetry components, plus the currently available instrumentation. Just to unpack it a bit, there are three critical packages, beyond the launcher itself, which are worth understanding as they explain how OpenTelemetry is structured.
- opentelemetry-api - the API package contains the OpenTelemetry instrumentation API. This package only contains interfaces, no implementation. It is safe to bring into any package without concern that a large dependency chain may follow it.
- opentelemetry-sdk -the SDK package contains the standard implementation for OpenTelemetry. This implementation is a framework written in python, allowing for various exporters, samplers, and lifecycle hooks to be plugged in, so that a wide variety of analysis tools can be supported.
- opentelemetry-instrumentation - this package contains two command line tools for automatically instrumenting your application:
opentelemetry-instrument. We’re going to use both of them now.
Install Automatic Instrumentation
The first command to learn is
opentelemetry-bootstrap. This will inspect the currently installed site-packages, and detect any packages we have instrumentation available for. By default, it prints out the packages to be copied into a requirements file, but it can also install them for you. For this example, let’s run it in installation mode.
And that’s it for installing OpenTelemetry!
The easiest way to run OpenTelemetry is via the
opentelemetry-instrument command, using env vars for configuration. You can find a list of available configurations here, but there are only two which are required:
- LS_SERVICE_NAME - The name for this type of service. We’ll use hello-sever and hello-client, respectively.
- LS_ACCESS_TOKEN - You can find this one by first logging into your Lightstep account (or creating one), then going to the settings page. Use the clipboard button to copy the access token.
To run OpenTelemetry with Lightstep, first log into your account (or create one) and find your access token on the Settings page. Use the clipboard button to copy the access token.
Do the same for the client.
Check for data by clicking on the Lightstep explorer (or equivalent). Yes! We see some spans. Click in to one and check out the trace.
Let’s pause for a second and review the data we are looking at. There are two spans, one from the requests package on the client, and one from the flask package on the server. Clicking on a span, you can see that it is already rich with data.
- http.* and net.* – these conventions describe everything about the request.
- instrumentation.name – this describes the instrumentation package which generated the span.
- span.kind - either client, server, or internal.
In OpenTelemetry, the representation of common concepts like HTTP are standardized across languages. As a result, analysis tools can automatically interpret the data they are looking at. We refer to these standardized attributes as semantic conventions. The complete list of can be found here.
No Code Required
The biggest, most important note is that we added OpenTelemetry to our service, but didn’t write any code. Everything could be done from the command line. This means that OpenTelemetry can potentially be added to a service by an operator, with a simple modification to deployment.
I highly recommend this approach as a first pass, before adding any additional detail. OpenTelemetry needs to be installed in every service in order for distributed tracing to work. It is more important to get every service instrumented at a high level than it is to dig in and deeply instrument the application code in a particular service. Library level instrumentation (flask, requests, redis, etc) will give you enough information to set up alerting root causing issues.
If this is the first time you’ve added distributed tracing to your system, don’t be surprised if a number of latency-related issues immediately become visible! After you’ve done a wide scale rollout, you can dig in selectively and add detail where needed. Converting your existing logs to span events is another great way to add detail without having to write a lot of code.
Using the OpenTelemery Python API
Okay, automation is great, but eventually you are going to want to add detail. Spans are already decorated with standardized attributes, but once you’re settled in, you will want to start adding more detail.
The most important details to add are application-level attributes critical to segmenting your data. For example, a projectID allows you to differentiate between errors that are affecting everyone connecting to a service, vs errors that are localized to a handful of accounts. Those would be two very different scenarios, and you would probably start looking in different places based on that feedback.
Also, logs. They are a thing. OpenTelemetry has a structured logging facility, we just call it events.
Adding Data to the Current Span
To add additional data to your trace, you need access to the currently active span. Since context propagation is already set up, thanks to the automatic instrumentation, there is already a span available to you.
Attributes are simply key value pairs. Events consist of a message, and a dictionary of attributes.
It’s best to add data to existing spans, rather than create child spans. This keeps all of the attributes grouped together, which makes for better indexing.
Creating a Child Span
Of course, you are going to want to create child spans on some occasions. A span represents a distinct operation - not an individual function, but an entire operation, such as a database query. Generally, this means you shouldn't be creating spans in your application code, they should be managed as part of the framework or library you are using.
But, that said, here is how you do it. First, create a tracer. A tracer is just a namespace - it let’s you know which package created the span, via the instrumentation.name attribute (you can also add a version as a second parameter).
Span management has two parts - the span lifetime and the span context. The lifetime is managed by starting the span with a tracer, and adding it to a trace by assigning it a parent.
Using spans directly like this is cumbersome. Instead, we want to create a new context where the span is active, so that it can be accessed by get_current_span instead of passing it around. In almost all cases, the easiest way to manage a span is by calling start_as_current_span.
If you ever need to create a span in your application code, I strongly recommend using the above pattern.
Ok, one final bit. We’ve covered spans, attributes, and events. But what about exceptions? Exceptions are reported as events, but they should be properly formatted. As a convenience, OpenTelemetry provides a record_exception method for capturing them correctly.
Uncaught exceptions are automatically recorded as errors.
And that is that. All you need to know to get started with tracing in Python.
Hopefully, it’s clear that If you stick with the above patterns, you can get a great deal of visibility with very little work. Of course, there are many more details and options; you can check out the API documentation for more information. I also have a more involved getting started guide; it works as a handy reference for all of the procedures described above.
OpenTelemetry is still in beta due to API changes, but it is also already in production across many organizations. If you stick to a Distro and automated instrumentation, you can use OpenTelemetry today without much fear of a breaking change affecting you.
Opinions expressed by DZone contributors are their own.