debugger VS opentelemetry-go

dlsniper/debugger: debugger middleware

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago (https://github.com/open-telemetry/opentelemetry-go/issues/67...).

I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are:

- Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch.

- Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle and tricky.

- Directly marshal protos instead of reflection with a hand-rolled library or with https://github.com/VictoriaMetrics/easyproto.

The gold standard is how TiDB implemented tracing (https://www.pingcap.com/blog/how-we-trace-a-kv-database-with...). Since Go purposefully (and reasonably) doesn't currently provide a comparable abstraction for thread-local storage, we can't implement similar tricks like special-casing when a trace is modified on a single thread.

OpenTelemetry GitHub Repository

When using Hono, you can leverage Fiberplane's client library, which instruments the application based on OpenTelemetry. This allows you to use Fiberplane Studio, which not only displays your endpoints and helps you make requests or replay your webhook but also captures the call chain (traces) among different integrations.

Many development teams leverage OpenTelemetry (OTeL) on the server side to collect signals such as logs, traces, and metrics from their applications. However, what's often overlooked is the power of OTeL's browser instrumentation. This client-side observability data can provide meaningful insights and create a comprehensive view of an application's performance by connecting traces between the client and server.

This is where things get complicated. Implementing such a system is a lot of work, and you should rely on upcoming standards like the W3C recommendation on trace contexts. You should rely on a standard because they solve two major pain points with observability: (1) they have popular implementations like OpenTelemetry (OTel) – which happens to be the second most active CNCF project to this day – and (2) they are widely adopted by major observability backends (e.g., Datadog).

The Vercel AI SDK now supports tracing with OpenTelemetry, an open-source standard for recording telemetry information, as an experimental feature. Here is an example of how trace visualization looks with the Vercel Datadog integration:

I'm not quite sure how to place this. For general app telemetry there's https://opentelemetry.io and for tracking user behavior similar to how it's explained in your docs it looks like https://segment.io would be the closest competitor.

How is telemetry.sh different from the above-mentioned?

Bruno from Inngest here, thanks for asking!

In general, we use OpenTelemetry[1] for instrumenting our services in production, collecting metrics and logs for important events. Specifically, we have set up

- multiple dashboards informing us about current system usage (events received, processed) including e2e latency distributions, compute resource usage for different deployments, and top operations

- metrics on critical systems (data stores including Redis, messaging infrastructure, connection poolers for Postgres, etc.) to gauge current resource utilization and typical load patterns

- alerting on unexpected deviations in KPIs (a subset of the metrics above) to help us spot and react to issues quickly

- forecasting on product usage and compute resource utilization patterns for planning medium to long-term infrastructure work

Hope this helps!

[1]: https://opentelemetry.io/

OpenTelemetry is the future of observability. It is being pushed by Google, Microsoft and Amazon to provide a vendor-neutral solution for traces, metrics and logs. The OpenTelemetry API implementation is available for many programming languages. I will only use metrics and the implementation for the language C# for this tutorial. Let's see how far apart reality and desire are.