trace-context-w3c VS opentelemetry-go

The answer is Context Propagation. The HTTP example is a classic and W3C even covers it. The propagation is adding the important fields from the context into the HTTP headers and having the other application extract those values and inject them into its trace context. This concept applies to any other way of communication. Here, we will focus on message brokers and how you can achieve context propagation for those.

I've been playing with OTEL for a while, with a few backends like Jaeger and Zipkin, and am trying to figure out a way to perform end to end timing measurements across a graph of services triggered by any of several events.

Consider this scenario: There is a collection of services that talk to one another, and not all use HTTP. Say agent A0 makes a connection to agent A1, this is observed by service S0 which triggers service S1 to make calls to S2 and S3, which propagate elsewhere and return answers.

If we limit the scope of this problem to services explicitly making HTTP calls to other services, we can easily use the Propagators API [1] and use X-B3 headers [2] to propagate the trace context (trace ID, span ID, parent span ID) across this graph, from the origin through to the destination and back. This allows me to query the metrics collector (Jaeger or Zipkin) using this trace ID, look at the timestamps originating at the various services and do a T_end - T_start to determine the overall time taken by one call for a round trip across all the related services.

However, this breaks when a subset of these functions cannot propagate the B3 trace IDs for various reasons (e.g., a service is watching a specific state and acts when the state changes). I've been looking into OTEL and other related non-OTEL ways to capture metrics, but it appears there's not much research into this area though it does not seem like a unique or new problem.

Has anyone here looked at this scenario, and have you had any luck with OTEL or other mechanisms to get results?

[1] https://opentelemetry.io/docs/specs/otel/context/api-propaga...

[2] https://github.com/openzipkin/b3-propagation

[3] https://www.w3.org/TR/trace-context/

-- https://www.w3.org/TR/trace-context/

For context propagation, why not just reuse the existing trace context that most frameworks and toolkits generate for http requests? I've had to apply some elbow grease to get it play nice but once it does you're able to use tools like Jeager, etc as part of your asynchronous flow as well.

(Note that OpenTelemetry uses, by default, the W3C context propagation specification, while Istio uses the B3 context propagation specification – this can be modified).

World Wide Web Consortium (W3C) has recommendations on the format of trace contexts. The aim is to develop a standardized format of passing trace context over standard protocols like HTTP. It saves a lot of time in distributed tracing implementation and ensures interoperability between various tracing tools.

The main objective is to propagate a message with traceparent id throw two APIs and one worker using W3C trace context standard. The first-api calls the second-api by a http call while the second-api has an asynchronous communication with the worker by a message broker (rabbitmq was chosen for that). Furthermore, zipkin was the trace system chosen (or vendor as the standard call it), being responsible for getting the application traces and building the distributed tracing diagram:

Our first approach was to implement a separate SDK for each independent technology stack. We decided to use OpenTelemetry which is widely adopted and covers most of our needs.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

OpenTelemetry

When choosing distributed tracing tools, considerations include your technology stack, business requirements, and monitoring complexity. Zipkin, SkyWalking, and OpenTelemetry are popular distributed tracing solutions, each with its unique features.

You can follow this process with any large token AI system like Claude by identifying tracing data relevant to the code you are working on, using it as context to prompt OpenAI or other LLMs. Generally, you’d generate tracing data by implementing OpenTelemetry (aka OTEL) libraries into your application, adding spans to your functions with Jaeger, or using commercial SaaS tools like Honeycomb and Datadog.

While many programming languages provide robust support for Open Telemetry, this instance focuses on Golang. It's important to note that, in the current context, the logs SDK for Golang is not implemented. For future reference consult the list of supported languages and explore the Open Telemetry repositories. Always prioritize the main repository and its contrib repository, housing extensions and instrumentation libraries crucial to the Open Telemetry framework. Stay updated with the latest developments to ensure seamless integration and enhanced functionality.

OneUptime (https://github.com/oneuptime/oneuptime) is the open-source alternative to DataDog. It's 100% free and you can self-host it on your VM / server / cloud or you can use SaaS at https://oneuptime.com

NEW UPDATES (since we last posted to HN): We now support OpenTelemetry (https://opentelemetry.io/) natively which will help you to monitor, observe and debug any app, service, database or stack.

OpenTelemetry is the fastest growing Cloud Native Computing Foundation (CNCF) project. It standardizes the instrumentation and collection of traces, metrics, and logs from applications, and is supported by all the major observability projects, languages, and tools. One standard to rule them all!

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. https://opentelemetry.io/

Tracetest uses your existing OpenTelemetry traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!