Tail-Based Sampling in the OpenTelemetry Collector: Keeping the Traces That Matter

Head-based sampling makes a decision the instant a trace starts, before anyone knows whether that trace is boring or the one you will spend Friday night chasing. That is the wrong time to decide. At that point the request has not failed yet, and the slow dependency call that will define it is still milliseconds away. Head sampling commits before any of that is visible, so it discards a random slice of exactly the traces you will later wish you had kept.

Tail-based sampling flips the order. It buffers the spans of a trace until the trace is complete, then decides once the errors and timing are actually on the record. The OpenTelemetry Collector ships a tail_sampling processor that does this well. It also has one operational trap that most tutorials skip, and getting it wrong quietly corrupts every decision the processor makes. This walks through a policy set that keeps the traces worth keeping, and then through the trap.

How the Processor Actually Works

The tail_sampling processor groups incoming spans by trace ID and holds them in memory. It waits decision_wait seconds for more spans in the same trace to arrive, then evaluates the buffered trace against your policies. If the decision is to sample, the whole trace is exported. Otherwise it is dropped.

A minimal configuration that keeps every error and a baseline of everything else:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

One thing to internalize early: policies are not first-match-wins. By default, every policy votes, and if any policy votes to keep, the trace is kept. The config above does not mean "errors, otherwise 5 percent." It means "keep all errors, and independently keep 5 percent of everything (including errors)." That OR behavior is usually what you want, but it surprises people who read the list top-down like an if/else. The one exception is an inverted or drop policy, which votes to drop and overrides the keep votes, though none of the policies here use that.

A Policy Set That Keeps What Matters

The point of tail sampling is to encode "interesting" in policy. In practice, four categories cover most of it: errors, slow requests, business-critical paths, and a low baseline so healthy traffic is still visible.

processors:
  tail_sampling:
    decision_wait: 15s
    num_traces: 200000
    expected_new_traces_per_sec: 10000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

      - name: critical-routes
        type: string_attribute
        string_attribute:
          key: http.route
          values:
            - /api/v1/checkout
            - /api/v1/payment

      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

This keeps every errored trace, every trace slower than a second, every trace through checkout or payment, and 2 percent of the rest. You can go further with numeric_attribute (keep transactions over a value, or traces with more than N database calls, a cheap way to catch N+1 queries), span_count (keep unusually complex traces), and and composite policies when a single condition is too blunt, for example "slow AND in production AND an API call." Reach for the composite policy when a plain latency rule would sweep in noise from batch jobs or health checks.

The Trap: A Trace Decided On Half Its Spans

Here is the part that breaks silently. The processor can only make a correct decision if it can see the whole trace. A trace is not correct or incorrect in isolation; a checkout trace might have twenty spans across six services. If those spans are split across two collector instances, each instance sees a fragment, evaluates a fragment, and decides on a fragment. The instance that never received the errored span happily drops the trace. You do not get an error. You get a slow, steady loss of exactly the traces your policies were written to keep, and it looks like the policies are just not matching.

A single collector sidesteps this, because it sees everything, but a single collector does not scale and is a single point of failure. The first time I ran into this, error traces started disappearing the day we scaled the sampling collector from one replica to three, and nothing alerted, because the fragments that survived still parsed as valid traces. The moment you run more than one tail_sampling instance, you have to guarantee that all spans of a given trace land on the same instance. The Collector solves this with a two-tier layout. A first tier receives spans and routes them by trace ID using the load_balancing exporter (older configs call it loadbalancing, now a deprecated alias). A second tier runs the actual tail_sampling processor.

Tier one, the router:

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-sampling.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [load_balancing]

The routing_key: traceID setting is the whole point. It hashes on trace ID so every span with the same trace ID is sent to the same downstream instance. The DNS resolver watches a headless service and keeps the backend list current as sampling pods come and go, rehashing when the set changes.

Tier two, the sampler, is a normal tail_sampling pipeline that receives the already-grouped spans and exports the survivors to your backend:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/backend]

Run tier two as a StatefulSet or a stable set of replicas behind that headless service. Put batch after tail_sampling, not before, so you are batching the survivors rather than shuffling spans ahead of the grouping.

Sizing Decision_wait and Memory

Two settings decide whether this is stable. decision_wait has to be longer than your slowest realistic trace, or you will evaluate traces before their tail spans arrive and drop good data. Rough starting points: 5 to 10 seconds for a monolith, 15 to 20 for microservices, 30 or more when traces cross regions. If you see "interesting" traces getting dropped, this is the first knob to turn.

num_traces is the in-memory buffer, and memory is the constraint people hit. A workable estimate:

num_traces        ≈ expected_new_traces_per_sec × decision_wait × 1.2
memory  ≈ average_trace_size × num_traces

At 10,000 traces per second, a 15-second wait, and 10 KB per trace, you are holding roughly 180,000 traces and around 1.8 GB before headroom. Size the pods for it and add 20 to 30 percent buffer, because an out-of-memory kill on a sampling collector drops whatever it was holding.

Confirm It Is Actually Working

Do not trust it because it started. The processor emits metrics that tell you the truth:

• otelcol_processor_tail_sampling_count_traces_sampled breaks down kept-versus-dropped by policy. If your errors policy is sampling almost nothing, either you have very few errors or your status codes are not set the way you think.
• otelcol_processor_tail_sampling_sampling_trace_removal_age is how old a trace is when it leaves the buffer. At steady state, it sits near decision_wait, and that is healthy: a trace waits, gets decided, and is removed. The warning sign is the opposite. If it drops well below decision_wait, the buffer is full, and traces are being evicted before they can be decided, so raise num_traces or add replicas.
• otelcol_processor_tail_sampling_sampling_decision_timer_latency shows how long decisions take. It is your early warning that the instance is overloaded.

The end-to-end check that matters: trigger a known error and a known slow request in a test environment, then confirm both traces show up complete in your backend. If they arrive whole, your routing is correct. If they arrive missing spans, the load-balancing tier is not doing its job, and every decision above it is suspect.

Tail sampling earns its place because it keeps the traces you will actually open: the failures and the slow paths, not a random 2 percent that probably misses both. But the processor is only as good as the traces it can see in one place. Get the trace-ID routing right first. The policies are the easy part.