Tracing KumuluzEE Microservices With Jaeger and Zipkin

Introduction

When it comes to developing applications within the microservice architecture, the number of microservices can grow quickly. Managing microservices becomes harder with each new or updated microservice. When an application experiences a slowdown and its "data flow" goes through several different microservices, pinpointing the exact location of a slowdown may be difficult for a developer.

Before starting with the actual code and tutorial portion, let us start with a brief explanation of what tracing is. Tracing is a way to track a request from its starting point through the entire network of microservices. In a typical Java microservice application, the "chain" starts either with the front-end or API making a REST request to some entry microservice. That microservice then handles the request (usually with JAX-RS) and queries other microservices, databases, or external applications. As we can see, the number of requests inside the network increases and spreads out throughout multiple different destinations (especially in large applications). This is all well and good, but when a problem occurs a developer must go through a lot of logs (produced by the selected logging framework, such as Fluentd or Logstash – in KumuluzEE you can use the KumuluzEE Logs extension to simplify logging) to find the problem. If a problem is severe (such as a complete crash), a solution is usually discovered quickly or even handled automatically by some container orchestration tool (such as Kubernetes). Tougher problems are the slowdowns in which the application still works but in a limited capacity.

This is where the tracing comes in. When implemented across the entire application, entire request flows are saved and presented to the user in an easy to use graphical interface. The base unit is called a span. A span can be an incoming or outgoing request; execution of a Java method; access to the database; etc. Each span contains basic information such as name and timestamps (additional data can be added within the code). Spans are related to each other (children, nested spans) and together form a "trace." More detailed analysis can be made by joining microservices and their relations to a graph. In conclusion, a trace is a sequence of events, called spans, which describe a path through the application.

KumuluzEE OpenTracing

KumuluzEE OpenTracing was released as an extension for the KumuluzEE framework. It is a part of MicroProfile specification. It supports both Jaeger and Zipkin.

We will demonstrate how to add tracing to an existing KumuluzEE application using the KumuluzEE OpenTracing extension. The whole process will be demonstrated step-by-step with screenshots of each step.

Prerequisites

Before starting, make sure, that you have the following things ready:

• Installed Java (8 and up).

• Installed Docker (for running Jaeger or Zipkin; this is optional if you are going to run Jaeger/Zipkin as a standalone service),

• Cloned starting project from GitHub (https://github.com/kumuluz/kumuluzee-samples/tree/master/kumuluzee-opentracing-tutorial).

Jaeger

Running Jaeger can be as simple as entering this line in the console:

$ docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.9

The Jaeger GUI is now accessible on http://localhost:16686. It consists of three main screens. The first one is the "Search" tab, which enables us to search through our traces with different criteria.

The second one is the "Compare" tab, which allows for a comparison of two spans.
The last one is the "Dependencies" tab, which displays a graph of our microservices and their respective connections (not shown due to being an empty page).

Right now, we have no data. Sample data can be added by exploring the GUI because Jaeger adds its own traces.

We can now open one trace.

Zipkin

If you have chosen to use Zipkin instead of Jaeger, Zipkin can also be run inside Docker:

docker run -d -p 9411:9411 openzipkin/zipkin

The Zipkin GUI is accessible on http://localhost:9411. The interface is very similar to Jaeger. The entry screen looks like this:
We can not test Zipkin yet due to not having any data (Zipkin does not store its own traces).

Starting Project Structure

Before diving deeper, let us explain our starting project.

The project consists of 5 microservices:

1. master - This is the entry point of the application. It is served on http://localhost:8080 (actual endpoint on v1/master). When queried, it makes two requests: to the alpha endpoint v1/alpha and to the beta endpoint v1/beta.

2. alpha - This is the first of 4 "slave" microservices. It is served on http://localhost:8081 and has two endpoints: /v1/alpha (just returns a value) and /v1/alpha/beta, which queries the gamma endpoint v1/gamma.

3. beta - This is the second of 4 "slave" microservices. It is served on http://localhost:8082 and has one endpoint: /v1/beta, which queries alpha endpoint /v1/alpha/beta. Simulated lag is added to this request (random delay).

4. gamma - This is the third of 4 "slave" microservices. It is served on http://localhost:8083 and has one endpoint: /v1/gamma, which queries delta endpoint /v1/delta. This microservice is different because it uses a simulated database with CDI.

5. delta - This is the last of 4 "slave" microservices. It is served on http://localhost:8084 and has one endpoint: /v1/delta, which just return a value.

Adding KumuluzEE OpenTracing Dependency

To start with tracing the first thing we need to do is add the KumuluzEE OpenTracing dependency to our application. Locate the pom.xml file in the root folder and add the following dependency for Jaeger:

<dependency>
    <groupId>com.kumuluz.ee.opentracing</groupId>
    <artifactId>kumuluzee-opentracing-jaeger</artifactId>
    <version>${kumuluzee-opentracing.version}</version>
</dependency>

For Zipkin:

<dependency>
    <groupId>com.kumuluz.ee.opentracing</groupId>
    <artifactId>kumuluzee-opentracing-zipkin</artifactId>
    <version>${kumuluzee-opentracing.version}</version>
</dependency>

This needs to be done for all microservices. At the time of writing this post, the latest version of KumuluzEE OpenTracing was 1.3.1. You do not need to add the version manually because the version is defined in the root pom as a variable and will be used automatically.

Just by adding this dependency, tracing is automatically enabled on all incoming JAX-RS requests. To see how this looks inside your chosen tracing GUI, simply visit the master endpoint in your browser:
http://localhost:8080/v1/master. After the page loads, we will see if any traces were added.

Commit for this step (for Jaeger)

We can see that there is a trace for each microservice with confusing names. That is not what we want, but it is a good first step.

The next step is to add the service name configuration field in order to make our traces more readable since the default service name is the Kumuluz instance id, which is just a bunch of numbers and letters (as seen in the image). We can do this in the config.yml file (located in src/main/resources). Add the service name field:

kumuluzee:
  opentracing:
    jaeger:
      service-name: master 
    zipkin:
      service-name: master

We can also add the property kumuluzee.name, which will be used as a service name if the service name is not provided (the service name is checked first, then kumuluzee.name and lastly the instance id). Remember, this needs to be done for all five microservices.

Let us rerun our microservices and try again. Traces are now more human-friendly and readable:

There are several other settings available, but we do not need them for this sample. For more information about other settings, check the KumuluzEE OpenTracing GitHub page.

Note: Settings for Jaeger and Zipkin are different. Make sure that you are setting the correct fields.

Commit for this step (for Jaeger)

Adding JAX-RS Outgoing Requests Tracing

As already mentioned before, incoming JAX-RS requests are traced automatically. The same does not apply to outgoing requests. At the moment, we have traces for each individual microservice, but we want to see the whole trace grouped together. We need to add some code to achieve that. First, locate the Resource.java file (src/main/java/com/kumuluz/ee/samples/opentracing/tutorial/master) and change the initialization logic for the  Client class.

private Client client = ClientBuilder.newClient();

with

private Client client = ClientTracingRegistrar.configure(ClientBuilder.newBuilder()).build();

For a microservice to resume the trace of some other microservice, the details of the trace need to be sent along with the REST request (achieved with HTTP headers). This is what a ClientTracingRegistrar.configure method does: it makes sure that the required headers are added to each outgoing request. After changing this line in all microservices (except the delta microservice, which does not have a client), the result should be the following trace:

If we open this trace, we can see the whole request with all the microservices in one.
This is exactly what we wanted; the overview of the whole request will all the timestamps. We can also look at the "Dependencies" tree now, which is updated with our microservices:

Commit for this step

Additional Features

We will demonstrate three additional features of tracing:

• Handling exceptions in spans.

• Adding custom data to spans.

• Adding custom spans (such as database access).

Handling Exceptions

Exceptions are handled automatically by KumuluzEE OpenTracing. To demonstrate this, we will throw an exception in the delta microservice. Locate the Resource.java file (src/main/java/com/kumuluz/ee/samples/opentracing/tutorial/delta) and add the following line:

@GET
public Response get() {
    throw new RuntimeException("Something went wrong here.");
    // return Response.ok("delta").build();
}

We now have to restart the delta microservice and refresh the page. The created trace now looks like this:

We can see from the trace that exception was added to the trace as a log. We will cover adding logs in the next section.

Adding Custom Data to Spans

If we look back to our project structure, we added some simulated lag to our application in the beta microservice. Basically, we added a random delay from 1 to 1000 milliseconds to the request. We will add this parameter to the trace to see how long we have to wait for the request. We start by moving the wait time to a new variable. Then we inject the Tracer instance (we also added @RequestScoped for CDI injection). After that, we can access the current span with tracer.activeSpan and add our delay to it. We can do it in three ways: with adding a tag, adding a log entry or adding a baggage item. We will demonstrate all three:

• setTag(); 

• log(); 

• setBaggageItem(); 

Full code (src/main/java/com/kumuluz/ee/samples/opentracing/tutorial/beta/Resource.java):

@Path("beta")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@RequestScoped
public class Resource {
    private Client client = ClientTracingRegistrar.configure(ClientBuilder.newBuilder()).build();

    @Inject
    private Tracer tracer;

    @GET
    public Response get() {
        try {
            int waitDelay = ThreadLocalRandom.current().nextInt(1, 1000 + 1);
            tracer.activeSpan().setTag("waitDelay", waitDelay);
            tracer.activeSpan().log("Waited " + waitDelay + " milliseconds.");
            tracer.activeSpan().setBaggageItem("waitDelay", String.valueOf(waitDelay));
            Thread.sleep(waitDelay);
            Response r1 = client
                    .target("http://localhost:8081/v1")
                    .path("alpha")
                    .path("beta")
                    .request()
                    .get();
            String response = r1.readEntity(String.class);
            return Response.ok("beta->" + response).build();
        } catch (Exception e) {
            return Response.serverError().build();
        }
    }
}

Choosing a method of storing custom data is up to the developer. Tags are often used for storing metadata information (such as IP addresses, span types, versions, etc.), logs are used for storing messages (such as exceptions) and baggage is used for storing data, which can be retrieved later with the getBaggageItem() method.

This is not the only thing we can do with an injected tracer. By injecting the tracer, you get access to its methods. The main thing you can do with it is start spans manually. You can read more in the OpenTracing documentation. Also, read the documentation for more details on when to use each method of adding custom data to spans (baggage, log, or tag). Let us restart the beta microservice and see what our trace looks like now:

Commit for this step

Adding Custom Spans

The final thing we will do in this guide add tracing to functions outside of JAX-RS. This way we can include methods and functions that are outside of a REST service to the distributed trace. Typical examples are calls to the database, calls to external applications using protocols other than REST services, and similar scenarios.

To demonstrate this, we will show how to add calls to the database. We have implemented a simulated database in our gamma microservice. The easiest way to add custom spans is to use the @Traced annotation and put it on the class. This way, all the methods will be traced. This annotation can also be used on a single method if we only want to trace a specific method inside the class. It is also possible to annotate the class and then disable tracing on methods by annotating them and setting the property value to false. Annotating and setting the value to false can also be used to disable automatic tracing of JAX-RS incoming requests.

We will put the @Traced annotation to our Database class and all methods inside the class will be traced. It is also possible to change the span name by changing the operationName parameter. Let us see how this looks like inside the code (Database.java file, located in src/main/java/com/kumuluz/ee/samples/opentracing/tutorial/gamma):

@ApplicationScoped
@Traced(value = true, operationName = "testingChangedOperationName")
public class Database {
    private HashMap<Integer, String> data;

    @PostConstruct
    private void init() {
        data = new HashMap<Integer, String>();
        data.put(1, "gamma");
    }

    public String get(Integer id) {
        return data.get(id);
    }

}

As a result, we get the following span added to our trace:
We could achieve the same thing by injecting a tracer and manually creating the span. This is used when a developer wants a more fine-grained control over created spans. That way, more that one span can be created inside one method.

Commit for this step

Summary

In this article, we have demonstrated the basic principles of distributed tracing for microservices. We implemented the tracing of an existing application using the KumuluzEE OpenTracing extension. We did not need to write a lot of code. We only changed a few lines and added some annotations. This shows how simple it is to add distributed tracing to your existing microservices and get all the benefits of distributed tracing such as tracing requests through an entire network of microservices and easier pinpointing of slowdowns.

Even though we covered the basics of tracing, we left a few things out. For example, we did not include integration with a MicroProfile Config extension and KumuluzEE config frameworks, which would allow additional tracing configuration such as ignoring tracing on JAX-RS endpoints and changing the way that spans are named. We have explained more advanced features only briefly but would suggest the reader dig deeper into OpenTracing if they plan to use it in real-world applications. We also used the basic Jaeger/Zipkin configuration and ran the all-in-one Docker images. Before running either one in production, consult the documentation for proper use cases and guidelines.

Disclaimer: this article is an updated version of the original with the only difference being the added Zipkin part.

Useful Links and References

For more information, visit the following links:

• KumuluzEE (https://ee.kumuluz.com/ and https://github.com/kumuluz/),

• KumuluzEE Blog (https://blog.kumuluz.com/),

• KumuluzEE OpenTracing (https://github.com/kumuluz/kumuluzee-opentracing),

• MicroProfile (https://microprofile.io/),

• MicroProfile OpenTracing (https://github.com/eclipse/microprofile-opentracing),

• Jaeger distributed tracing (https://www.jaegertracing.io/),

• Zipkin distributed tracing (https://zipkin.io/),

• OpenTracing (https://opentracing.io/),

• Original blog post (https://blog.kumuluz.com/developers/community/2019/02/11/kumuluzee-opentracing-jaeger).