opentelemetry-dotnet VS trace-context-w3c

open-telemetry/opentelemetry-dotnet: The OpenTelemetry .NET Client (github.com)

💡Good to know -- If you wish to export traces to Jaeger, you should use the AddJaegerExporter instead of the AddOtlpExporter. Visit the opentelemetry-dotnet repository to see how it's done.

We're going to use OpenTelemetry .NET SDK. Add following nuget dependencies to the project:

We instrument the application with the OpenTelemetry SDK and SqlClient instrumentation library for .NET. First, we add the following NuGet package references to the API’s project file:

Besides that, the propagation fields (traceparent and tracestate) were added in the message header. In the last article, I said that the standard (in the Working Draft (WD) step of the w3c process) recommends to add the propagation fields in the application-properties section by the message publisher. For the current example, I chose to propagate that context in the message header even for AMQP calls as was done in the dotnet OpenTelemetry example. It's important to reinforce that Trace Context: AMQP protocol is not a W3C Recommendation yet. Take a look at the place where the propagation fields were added:

As long as the Azure Functions app knows the instrumentation key from an Application Insights instance, it traces almost everything. OpenTelemetry.NET is one of the Open Telemetry implementations, has recently released v1.0 for tracing. Both metrics and logging are close to GA. However, it doesn't work well with Azure Functions. Therefore, in this post, let's manually implement the tracing at the log level, which is sent to Application Insights.

The answer is Context Propagation. The HTTP example is a classic and W3C even covers it. The propagation is adding the important fields from the context into the HTTP headers and having the other application extract those values and inject them into its trace context. This concept applies to any other way of communication. Here, we will focus on message brokers and how you can achieve context propagation for those.

I've been playing with OTEL for a while, with a few backends like Jaeger and Zipkin, and am trying to figure out a way to perform end to end timing measurements across a graph of services triggered by any of several events.

Consider this scenario: There is a collection of services that talk to one another, and not all use HTTP. Say agent A0 makes a connection to agent A1, this is observed by service S0 which triggers service S1 to make calls to S2 and S3, which propagate elsewhere and return answers.

If we limit the scope of this problem to services explicitly making HTTP calls to other services, we can easily use the Propagators API [1] and use X-B3 headers [2] to propagate the trace context (trace ID, span ID, parent span ID) across this graph, from the origin through to the destination and back. This allows me to query the metrics collector (Jaeger or Zipkin) using this trace ID, look at the timestamps originating at the various services and do a T_end - T_start to determine the overall time taken by one call for a round trip across all the related services.

However, this breaks when a subset of these functions cannot propagate the B3 trace IDs for various reasons (e.g., a service is watching a specific state and acts when the state changes). I've been looking into OTEL and other related non-OTEL ways to capture metrics, but it appears there's not much research into this area though it does not seem like a unique or new problem.

Has anyone here looked at this scenario, and have you had any luck with OTEL or other mechanisms to get results?

[1] https://opentelemetry.io/docs/specs/otel/context/api-propaga...

[2] https://github.com/openzipkin/b3-propagation

[3] https://www.w3.org/TR/trace-context/

-- https://www.w3.org/TR/trace-context/

(Note that OpenTelemetry uses, by default, the W3C context propagation specification, while Istio uses the B3 context propagation specification – this can be modified).

World Wide Web Consortium (W3C) has recommendations on the format of trace contexts. The aim is to develop a standardized format of passing trace context over standard protocols like HTTP. It saves a lot of time in distributed tracing implementation and ensures interoperability between various tracing tools.

The main objective is to propagate a message with traceparent id throw two APIs and one worker using W3C trace context standard. The first-api calls the second-api by a http call while the second-api has an asynchronous communication with the worker by a message broker (rabbitmq was chosen for that). Furthermore, zipkin was the trace system chosen (or vendor as the standard call it), being responsible for getting the application traces and building the distributed tracing diagram:

If you got until here and liked the article content, let me know reacting to the current post. You can also open a discussion below, I'll try to answer soon. On the other hand, if you think that I said something wrong, please open an issue in the article's github repo. In the next article, I'll show a full distributed trace example using the trace context concept (in a microsservice architecture using .NET 5). Hope you like it!