dd-trace-rb VS opentelemetry-collector-contrib

This action starts two Ruby on Rails APIs, one instrumented with ddtrace and another with OpenTelemetry SDK, both connecting to an OpenTelemetry Collector that sends data to Jaeger:

Thing is, once you have 1) and 2), the added complexity of bringing in, integrating, and writing for a different tool to achieve 3) begins to make little sense, when you can just go along and do it just as well in rspec anyway... It's a matter of balance and heavily depends on the project.

> if you're still at Datadog

As a matter of fact I am. Feel free to shoot me an email.

    curl -s https://github.com/DataDog/dd-trace-rb/commit/176c642ca73679cabc5fa1a113bc9b600aa04dcd.patch | grep '^From:'

> For myself, I'm fine with the typing being in a separate .rbs file

We type[0] by having one separate .rbs file per .rb file. Works really well with an editor's vertical splits: type outline on one side, code on the other. That, or use something like vim-projectionist[1].

[0]: (WIP: there's a huge codebase to type, but we're progressively getting there) https://github.com/DataDog/dd-trace-rb/tree/master/sig

[1]: https://github.com/tpope/vim-projectionist

Thanks! I'll pass it on to the team :D

I've got to say, the folks at Intercom made it particularly fun. They were sending us traces and graphs from their internal systems when we trying to figure out some issues with them (e.g. we ran into this datadog context problem: https://github.com/DataDog/dd-trace-rb/issues/1389)

The OpenTelemetry Collector provides several processors out of the box. The most commonly used ones for compliance are:

https://github.com/open-telemetry/opentelemetry-collector-co...

Great questions!

OTel Metrics: I get it, it's specified as almost a superset of everyone's favorite metric standards with config for push/pull, monotonic vs delta, exponential/"native" histograms, etc. I have my preferences as well which would be a subset of the standard but I get why a unifying standard needed to be flexible.

Statsd: The great thing about the OTel collector is that it allows ingesting a variety of different data formats, so you can take in statsd and output OTel or write directly to ClickHouse: https://github.com/open-telemetry/opentelemetry-collector-co...

We correlate across trace/span id as well as resource attributes. The correlation across logs/traces with span/trace id is a pretty well worn path across our product. Metrics to the rest is natively done via resource attributes and we primarily expose correlation for K8s-based workloads with more to come. We don't do exemplars _yet_ to solve the more generic correlation case for metrics (though I don't think statsd can transmit exemplars)

Elixir: We try to do our best to support wherever our users are, the OTel SDK and ours have continued to change in parallel over time - we'll want to likely re-evaluate if we should start pointing towards the base OTel SDK for Elixir. We've been pretty early on the OTel SDK side across the board so things continue to evolve, for example our Deno OTel integration came out I think over a year before Deno officially launched one with native HyperDX documentation <3

Notebooks: Yes, it should land in an experimental state shortly, stay tuned :) There's a lot of exciting workflows we're looking to unlock with notebooks as well. If you have any thoughts in this direction, please let me know. I'd love to get more user input ahead of the first release.

While the batch and memory processors are part of the core collector repository, the resource processor is part of the contrib collection. The resource processor is particularly useful for adding common metadata to all telemetry data passing through the collector. For example, we can use it to amend our logs with the service name and environment to enrich the information that is sent to the backend. You can read more about it here.

To be fair, the setup of the article works with most modern observability solutions, in same cases just by replacing the endpoint and authentication token. Turning telemetry processing into a sort of utility is one of the great things that OpenTelemetry did. Now, among vendors, we compete on delivering insights on the telemetry, as opposed to just collecting it. If you are interested, I wrote about it a while back [1].

About excessive telemetry, that depends on what you want to achieve. Using facilities in the OpenTelemetry Collector like [2], you can easily drop all telemetry you have no use for. At the cost of tooting my own horn, we actually provide super easy ways of doing the same dropping at no charge whatsoever to the end user in Dash0 [3].

[1] https://thenewstack.io/is-otel-the-last-observability-agent-...

[2] https://github.com/open-telemetry/opentelemetry-collector-co...

[3] https://www.dash0.com/changelog/spam-filters

> having a decent way to get metrics from logs ad-hoc completely solves the metric cardinality explosion.

last i checked, the span metrics connector[1] was supposed to "solve" this in otel; but i'm not particularly inclined, as configurations are fixed.

any data analytics platform worth it's money should be able to do this at runtime (for specified data volume constraints, in reasonable time).

in general, structured logging should also help with this; as much as i love regex, i do not think extracting "data" from raw logs is lossless.

[1] https://github.com/open-telemetry/opentelemetry-collector-co...

b)Deploy the OTEL Exporter (client) The OTEL Exporter is the client that generates (i.e source) the metrics. We use the hostmetrics receiver plugin. This will vary according to the needs. The other receiver is deployed in the client namespace. The client certificates are mounted as volumeMounts just as before.

You can also find the full OTel configuration directly in the Logz.io platform, under Integrations and searching for OpenTelemetry or accessing the Logz.io exporter GitHub documentation.

Collector Contrib Image: I referenced the Collector contrib image, which includes Azure Monitor as an available exporter.