trace-context-w3c VS apisix

The answer is Context Propagation. The HTTP example is a classic and W3C even covers it. The propagation is adding the important fields from the context into the HTTP headers and having the other application extract those values and inject them into its trace context. This concept applies to any other way of communication. Here, we will focus on message brokers and how you can achieve context propagation for those.

I've been playing with OTEL for a while, with a few backends like Jaeger and Zipkin, and am trying to figure out a way to perform end to end timing measurements across a graph of services triggered by any of several events.

Consider this scenario: There is a collection of services that talk to one another, and not all use HTTP. Say agent A0 makes a connection to agent A1, this is observed by service S0 which triggers service S1 to make calls to S2 and S3, which propagate elsewhere and return answers.

If we limit the scope of this problem to services explicitly making HTTP calls to other services, we can easily use the Propagators API [1] and use X-B3 headers [2] to propagate the trace context (trace ID, span ID, parent span ID) across this graph, from the origin through to the destination and back. This allows me to query the metrics collector (Jaeger or Zipkin) using this trace ID, look at the timestamps originating at the various services and do a T_end - T_start to determine the overall time taken by one call for a round trip across all the related services.

However, this breaks when a subset of these functions cannot propagate the B3 trace IDs for various reasons (e.g., a service is watching a specific state and acts when the state changes). I've been looking into OTEL and other related non-OTEL ways to capture metrics, but it appears there's not much research into this area though it does not seem like a unique or new problem.

Has anyone here looked at this scenario, and have you had any luck with OTEL or other mechanisms to get results?

[1] https://opentelemetry.io/docs/specs/otel/context/api-propaga...

[2] https://github.com/openzipkin/b3-propagation

[3] https://www.w3.org/TR/trace-context/

-- https://www.w3.org/TR/trace-context/

For context propagation, why not just reuse the existing trace context that most frameworks and toolkits generate for http requests? I've had to apply some elbow grease to get it play nice but once it does you're able to use tools like Jeager, etc as part of your asynchronous flow as well.

(Note that OpenTelemetry uses, by default, the W3C context propagation specification, while Istio uses the B3 context propagation specification – this can be modified).

World Wide Web Consortium (W3C) has recommendations on the format of trace contexts. The aim is to develop a standardized format of passing trace context over standard protocols like HTTP. It saves a lot of time in distributed tracing implementation and ensures interoperability between various tracing tools.

The main objective is to propagate a message with traceparent id throw two APIs and one worker using W3C trace context standard. The first-api calls the second-api by a http call while the second-api has an asynchronous communication with the worker by a message broker (rabbitmq was chosen for that). Furthermore, zipkin was the trace system chosen (or vendor as the standard call it), being responsible for getting the application traces and building the distributed tracing diagram:

Through this intelligent caching mechanism, APISIX efficiently utilizes system resources when handling a large volume of requests, thereby improving overall system performance and stability. APISIX, with its advanced LRU cache, provides developers with a reliable and efficient API gateway solution, facilitating smooth communication with external services.

The main issue is that priority is documented in the config-default.yaml file, while the phase is buried in the code. Worse, some plugins run across different phases. For example, let's check the proxy proxy-rewrite plugin and, more precisely, the functions defined there:

// references: // https://github.com/tetratelabs/proxy-wasm-go-sdk/tree/main/examples // https://github.com/apache/apisix/blob/master/t/wasm/ package main import ( "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm" "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types" "github.com/valyala/fastjson" ) func main() { proxywasm.SetVMContext(&vmContext{}) } // each plugin has its own VMContext. // it is responsible for creating multiple PluginContexts for each route. type vmContext struct { types.DefaultVMContext } // each route has its own PluginContext. // it corresponds to one instance of the plugin. func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext { return &pluginContext{} } type header struct { Name string Value string } type pluginContext struct { types.DefaultPluginContext Headers []header } func (ctx *pluginContext) OnPluginStart(pluginConfigurationSize int) types.OnPluginStartStatus { data, err := proxywasm.GetPluginConfiguration() if err != nil { proxywasm.LogErrorf("error reading plugin configuration: %v", err) return types.OnPluginStartStatusFailed } var p fastjson.Parser v, err := p.ParseBytes(data) if err != nil { proxywasm.LogErrorf("error decoding plugin configuration: %v", err) return types.OnPluginStartStatusFailed } headers := v.GetArray("headers") ctx.Headers = make([]header, len(headers)) for i, hdr := range headers { ctx.Headers[i] = header{ Name: string(hdr.GetStringBytes("name")), Value: string(hdr.GetStringBytes("value")), } } return types.OnPluginStartStatusOK } // each HTTP request to a route has its own HTTPContext func (ctx *pluginContext) NewHttpContext(contextID uint32) types.HttpContext { return &httpContext{parent: ctx} } type httpContext struct { types.DefaultHttpContext parent *pluginContext } func (ctx *httpContext) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action { plugin := ctx.parent for _, hdr := range plugin.Headers { proxywasm.ReplaceHttpResponseHeader(hdr.Name, hdr.Value) } return types.ActionContinue }

API7 takes Apache APISIX as its robust foundation, which is open-source and has an active community with over 600 contributors all over the world. The nature of open source allows users to examine the source code, which promotes transparency. This transparency helps users understand how APISIX works, verify its security, and identify and fix any potential vulnerabilities or bugs.

But the best part is that the libraries mentioned here and Apache APISIX are entirely open source, meaning you can look under the hood and modify things yourself.

Default configuration

4. Plugin definition: It is a really important part of plugin implementation that we define as a table with properties for the version, priority, name, and schema. The name and schema are the plugin's name and schema defined earlier. The version and priority are used by APISIX to manage the plugin. The version typically refers to the version that is currently in use like API versioning. If you publish and update your plugin logic, it is going to be 1.1 (You can set any version you wish). But you need to be very careful in choosing priority. The priority field defines in which order and phase your plugin should be executed. For example, the 'ip-restriction' plugin, with a priority of 3000, will be executed before the 'example-plugin', which has a priority of 0. This is due to the higher priority value of the 'ip-restriction' plugin. If you're developing your own plugin, make sure that you followed the order of plugins not to mess up the order of existing plugins. You can check the order of existing plugins in the config-default.yaml file and open the Apache APISIX Plugin Development Guide to determine.

Their use of etcd was a hard pass for me; I don't need more etcd in my life

Apache APISIX Apache APISIX is an open source, dynamic, real-time, high-performance cloud native API gateway. APISIX provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more. Official website https://apisix.apache.org/ GitHub projects APISIX (the core): https://github.com/apache/apisix GitHub - apache/apisix: The Cloud-Native API Gateway GitHub - apache/apisix-dashboard: Dashboard for Apache APISIX GitHub - apache/apisix-website: Apache APISIX Website GitHub - apache/apisix-docker: the docker for Apache APISIX GitHub - apache/apisix-go-plugin-runner: Go Plugin Runner for APISIX GitHub - apache/apisix-java-plugin-runner: APISIX Plugin Runner in Java GitHub - apache/apisix-python-plugin-runner: Apache APISIX Python plugin runner GitHub - apache/apisix-helm-chart: Apache APISIX Helm Chart GitHub - apache/apisix-ingress-controller: ingress controller for K8s

Grafana configuration. Most of it comes from the configuration provided by APISIX.