opentelemetry-examples VS opentelemetry-specification

We focused on the observability ecosystem and took the time to interact with our friends from Lightstep, New Relic, Honeycomb, Dynatrace, Instana, and many more. With that in mind, keep an eye out for more integrations coming to Tracetest!

Lightstep bills itself as a platform for the reliability of cloud-native applications. The people behind Lightstep co-founded OpenTelemetry and OpenTracing, which gives them a unique perspective on the use cases of distributed tracing and the value of having a vendor-neutral tracing data format.

In the last 5 to 10 years, new Observability vendors have entered the market, including Honeycomb, Instana, Lightstep and Datadog. Similarly, traditional APM vendors such as Dynatrace, AppDynamics, and New Relic, as well as SIEM (and log management) vendors such as Splunk and Sumo Logic, have joined them in the Observability space too. Finally you also have major cloud providers such as AWS with their own observability solution. Each of them is attempting to address the observability issues that modern architecture presents by using Logs, Metrics, Traces, and Events.

I spent Day 2 at the Colony Club to attend OTel Unplugged. This event was sponsored by Lightstep, Honeycomb, New Relic, Splunk, Dynatrace, Crowdstrike, and NGINX. I came into the event not knowing what to expect. I can sometimes clamp up when I’m around folks that I don’t know, but because I was helping with the event check-in, I got to say hello to a number of the attendees, which helped break the ice. And it turns out that there were a lot of names that I recognized from my work in the OTel community, and it was nice to connect in person with folks whom I’d only previously met through Slack or Zoom.

The example in this tutorial can be found in the lightstep/opentelemetry-examples repo. We will be working with three main files:

Note how we don’t have to set a LS_ACCESS_TOKEN, since that’s already configured in the Collector’s config.yml file. Just make sure that you have a running OTel Collector instance!

Note: If you’re looking for full code listings, don’t panic! You see them in the Lightstep OTel examples repository.

It means that the OpenTelemetry project provides not only a specification to define the contract between the applications, collectors, and telemetry databases, but also a set of APIs, SDKs, and tools like instrumentation libraries (for different languages), collectors, operators, etc. OpenTelemetry is open-source and vendor-agnostic, so the project is not tied to any specific vendor or cloud provider.

Sure, happy to provide more specifics!

Our main issue was the lack of a synchronous gauge. The officially supported asynchronous API of registering a callback function to report a gauge metric is very different from how we were doing things before, and would have required lots of refactoring of our code. Instead, we wrote a wrapper that exposes a synchronous-like API: https://gist.github.com/yolken-airplane/027867b753840f7d15d6....

It seems like this is a common feature request across many of the SDKs, and it's in the process of being fixed in some of them (https://github.com/open-telemetry/opentelemetry-specificatio...)? I'm not sure what the plans are for the golang SDK specifically.

Another, more minor issue, is the lack of support for "constant" attributes that are applied to all metrics. We use these to identify the app, among other use cases, so we added wrappers around the various "Add", "Record", "Observe", etc. calls that automatically add these. (It's totally possible that this is supported and I missed it, in which case please let me know!).

Overall, the SDK was generally well-written and well-documented, we just needed some extra work to make the interfaces more similar to the ones were were using before.

OpenTelemetry is an open-source collection of tools, APIs, and SDKs that aims to standardize the way we generate and collect telemetry data. It follows a specification-driven development. The OpenTelemetry specification  has design and implementation guidelines for how the instrumentation libraries should be implemented. In addition, it provides client libraries in all the major programming languages that follow the specification.

Two problems with OpenTelemetry:

1. It doesn't know what the hell it is. Is it a semantic standard? Is a protocol? It is a facade? What layer of abstraction does it provide? Answer: All of the above! All the things! All the layers!

2. No one from OpenTelemetry has actually tried instrumenting a library. And if they have, they haven't the first suggestion on how instrumenters should actually use metrics, traces, and logs. Do you write to all three? To one? I asked this question two years ago, not a single response. [1]

[1] https://github.com/open-telemetry/opentelemetry-specificatio...

OpenTelemetry Specification GitHub

One update to this: we proposed replacing the count suffix in OpenTelemetry with total to match Prometheus/OpenMetrics. That discussion resulted in the count suffix being removed from the OpenTelemetry semantic conventions. We'll soon update our metric from being called function.calls.count to just function.calls and the generated Prometheus queries will refer to function_calls_total. That resolves one of the main conflicts between the two specs.

This is your best bet if you want to track status updates: https://github.com/open-telemetry/opentelemetry-specification/issues/2911

OpenTelemetry is a standard for implementing telemetry in your applications. It provides a specification, containing the requirements that all implementations should follow as well as some implementations for major languages, including an API and a SDK to interact with it.

At this point, we are about to do the real work where we have to configure OpenTelemetry and export telemetry data to New Relic. Exporting this kind of data relies on a specific protocol; the OpenTelemetry Protocol or OTLP.

OpenTelemetry provides instrumentation libraries for your application. The development of these libraries is guided by the OpenTelemetry specification. The OpenTelemetry specification describes the cross-language requirements and design expectations for all OpenTelemetry implementations in various programming languages.