Running Jaeger Agent on Bare Metal

The Jaeger Agent is the component that is responsible for receiving the spans from the instrumented application and forwarding it to the Jaeger Collector, so that the data gets appropriately stored. In addition to acting as a span buffer between the application and the collector, the Jaeger Agent also receives updates from the collector regarding the sampling strategies, making said strategies available via a REST endpoint that is queried by the Jaeger Client, deployed within the instrumented application.

The Jaeger Client typically sends spans via UDP to the agent, avoiding the TCP overhead and reducing the CPU and memory pressure upon the instrumented application. With that in mind, the Jaeger Agent should be deployed as close as possible to the instrumented application, reducing the risks inherent to UDP delivery. A second aspect to take into consideration is the tenancy model, as each Jaeger Agent is typically used by a single tenant.

With the single-tenant scenarios, Jaeger Agent instances can be shared among multiple instrumented applications. This means that the agent runs as a daemon process in the same machine as the applications.

With the multi-tenant scenarios, there should be one Jaeger Agent per tenant, meaning that there are multiple agent daemon processes running, one per tenant.

The following SystemDservice unit file can be used to control the Jaeger Agent lifecycle. The example uses a YAMLconfiguration file located at /etc/jaeger-agent.yamland assumes that the Jaeger Agent binary is located at /usr/local/bin/agent-linux.

[Service]
ExecStart=/usr/local/bin/agent-linux --config-file=/etc/jaeger-agent.yaml
Restart=always
StandardOutput=syslog
SyslogIdentifier=jaeger-agent
User=nobody
Group=nobody

[Install]
WantedBy=multi-user.target

With that in place, the Jaeger Agent can be started by issuing the following command:

systemctl start jaeger-agent

The status and logs can be queried like any other process managed by SystemD, like:

systemctl status jaeger-agent

An empty configuration file at /etc/jaeger-agent.yamlis sufficient to enable an Agent to properly start with a collector running on localhost. On production environments, it’s recommended to have the collectors running as a cluster in their own hosts. A typical configuration file would, therefore, include at least the location of the Jaeger Collector, such as:

reporter:
  type: tchannel
  tchannel:
    host-port: jaeger-collector:14267

In a multi-tenant scenario, the “instantiated service” feature of SystemDis suitable. In that case, the unit file would look like:

[Service]
ExecStart=/usr/local/bin/agent-linux --config-file=/etc/jaeger-agent-%i.yaml
Restart=always
StandardOutput=syslog
SyslogIdentifier=jaeger-agent-%i
User=nobody
Group=nobody

[Install]
WantedBy=multi-user.target

Given a configuration file located at /etc/jaeger-agent-tenant1.yaml, start the agent for the tenant tenant1with:

systemctl start jaeger-agent@tenant1

As a good practice, explicitly set the host-port properties the agent should bind, like the following:

http-server:
  host-port: ":5778"
processor:
  jaeger-binary: 
    server-host-port: ":6832"
  jaeger-compact:
    server-host-port: ":6831" 
  zipkin-compact:
    server-host-port: ":5775"

The second tenant should then bind to different ports, like:

http-server:
  host-port: ":6778"
processor:
  jaeger-binary: 
    server-host-port: ":7832"
  jaeger-compact:
    server-host-port: ":7831" 
  zipkin-compact:
    server-host-port: ":6775"
reporter:
  type: tchannel
  tchannel:
    host-port: jaeger-collectors-tenant-2:14267

Starting the second tenant with systemctl start jaeger-agent@tenant2, there should be two Jaeger Agent instances running, each on its own set of ports. The example above also uses a different hostname for the target collector, used exclusively to handle spans from our second tenant.

We’ve seen here how we can make use of SystemD to manage our Jaeger Agent process, both with a single and multi-tenant scenario. With some simple changes, the same strategy can be used to manage Jaeger Collector and Jaeger Query processes.

How to deploy in Kubernetes and OpenShift is a topic for a future blog post. In the meantime, take a look at the Jaeger Operator, which handles most of the process for you.