DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Monoliths to Microservices: Untangling Your Spaghetti
  • Less Time Learning, More Time Building
  • AI-Driven API and Microservice Architecture Design for Cloud
  • Provision Cloud Infrastructure Using Google Duet AI

Trending

  • Hybrid Cloud vs Multi-Cloud: Choosing the Right Strategy for AI Scalability and Security
  • The Cypress Edge: Next-Level Testing Strategies for React Developers
  • How to Practice TDD With Kotlin
  • While Performing Dependency Selection, I Avoid the Loss Of Sleep From Node.js Libraries' Dangers
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Transforming Proprietary Trace Context to W3C Trace Context

Transforming Proprietary Trace Context to W3C Trace Context

GCP/AWS proprietary trace context to W3C standard trace context for out-of-the-box OpenTelemetry integration to support distributed tracing.

By 
Arvind Bharti user avatar
Arvind Bharti
·
Upasana Sharma user avatar
Upasana Sharma
·
May. 29, 24 · Tutorial
Likes (6)
Comment
Save
Tweet
Share
4.7K Views

Join the DZone community and get the full member experience.

Join For Free

Microservices-based architecture splits applications into multiple independent deployable services, where each service provides a unique functionality. Every architectural style has pros and cons. One of the challenges of micro-service architecture is complex debugging/troubleshooting.

Distributed Tracing

In a microservice world, distributed tracing is the key to faster troubleshooting. Distributed tracing enables engineers to track a request through the mesh of services and therefore help them troubleshoot a problem.

To achieve this, a unique identifier, say, trace-id, is injected right at the initiation point of a request, which is usually an HTTP load balancer. As the request hops through the different components (third-party apps, service-mesh, etc.), the same trace-id should be recorded at every component. This essentially requires propagation of the trace-id from one hop to another. 

Over a period, different vendors adopted different mechanisms to define the unique identifier (trace-id), for example:

  • Zipkin B3 headers
  • Datadog tracing headers
  • Google proprietary trace context
  • AWS proprietary trace context
  • Envoy request id
  • W3C trace context

An application can adopt one of these available solutions as per the need. Accordingly, the relevant header (e.g., x-cloud-trace-context if Google proprietary trace context is adopted) should get injected at the request initiation and thereafter same value should get propagated to each of the components involved in the request lifecycle to achieve distributed tracing.

W3C Trace Context Standard

As the microservice world is evolving, there is a need to have a standard mechanism for trace propagation. Consider the case when two different applications that adopted two different trace propagation approaches, are used together. Since they use two different headers for trace propagation, distributed tracing gets broken when they communicate. 

To address such problems, it is recommended to use the W3C trace context across all components. W3C trace context is the standard that is being adopted by all major vendors for supporting cross-vendor distributed traces.

Problem: Broken Traces

OpenTelemetry supports the W3C trace context header "traceparent" propagation using auto-instrumentation. This means, as an application developer, I need not write any code in my application for trace context propagation when I instrument it with OpenTelemetry. For example, if I have a Java application, I can instrument it as shown below:

java -javaagent:opentelemetry-javaagent.jar -Dotel.service.name=app-name -jar app.jar 

The traceparent header will now be automatically generated/propagated by the instrumented Java application.

However, when my application, instrumented using OpenTelemetry, gets deployed behind GCP or AWS HTTP load balancer, my expectation to visualize the complete trace starting from the load balancer fails. This is because

  • GCP HTTP Load Balancer supports their proprietary trace context header "X-Cloud-Trace-Context". See GCP documentation for more details.
  • AWS Elastic Load Balancer supports their proprietary trace context header "X-Amzn-Trace-Id". See AWS documentation for more details.
  • My application generates and logs the W3C traceparent header. 

So, the unique-identifier generated by the GCP/AWS load balancer is not propagated further by my application. This is the typical problem of broken traces, also described above.

So how can a developer leverage the out-of-the-box OpenTelemetry trace context propagation functionality?

Solution: GCP Trace Context Transformer

We have solved this problem by transforming the GCP/AWS proprietary trace context header (X-Cloud-Trace-Context/ X-Amzn-Trace-Id) to the W3C trace context header (traceparent).

Service mesh is a key component in a distributed system to enforce organization policies consistently across all the applications. One of the popular service mesh, Istio, can help in solving our problem. The diagram below elaborates on the solution:

traceparent

A common Trace-Id value across all the logs generated from the load balancer, istio-ingress gateway, istio-sidecar, and application logs helps in stitching all the logs for a request processing.

Istio allows you to extend the data-plane behavior by writing custom logic using either Lua or WASM.

We have extended the istio-ingress gateway by injecting a Lua filter. This filter extracts the trace-id and span-id from X-Cloud-Trace-Context and creates the traceparent request header using these values. 

Note: 

  • For the sake of simplicity, below filter code is built only for the GCP "X-Cloud-Trace-Context". One can write a similar filter for AWS "X-Amzn-Trace-Id".
  • While adopting the filter in your infrastructure, don't forget to choose the right namespace and workloadSelector label.
  • This filter has been tested on Istio 1.20.1 version.
YAML
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gcp-trace-context-transformer-gateway
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER # http connection manager is a filter in Envoy
    match:
      context: GATEWAY
    patch:
      operation: INSERT_BEFORE
      value: 
       name: envoy.filters.http.lua
       typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            function envoy_on_request(request_handle)

              local z = request_handle:headers():get("traceparent")
              if z ~= nil then
                return
              end

              local x = request_handle:headers():get("X-Cloud-Trace-Context")
              if x == nil then
                return
              end

              local y = string.gmatch(x, "%x+")
              local traceid = y()
              if (traceid == nil) then
                return
              end

              -- generate a new 16 hex-character random span
              math.randomseed(os.time())
              local parentid = math.random(0xffffffff, 0x7fffffffffffffff)
              local traceparent = string.format("00-%s-%016x-01", traceid, parentid)
              request_handle:headers():add("traceparent", traceparent)

            end

            function envoy_on_response(response_handle)
              return
            end


Alternate Solution: Custom GCP Trace Context Propagator

Another possible solution could be extending OpenTelemetry to support the propagation of GCP proprietary trace context. One implementation exists on GitHub but, alas, it is still in alpha state (at the time of publishing this article). Further, this solution will only work for GCP environments, similar propagators will be needed for different cloud providers (AWS, etc).

Cloud Load balancing (computing) microservice

Opinions expressed by DZone contributors are their own.

Related

  • Monoliths to Microservices: Untangling Your Spaghetti
  • Less Time Learning, More Time Building
  • AI-Driven API and Microservice Architecture Design for Cloud
  • Provision Cloud Infrastructure Using Google Duet AI

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!