hyperloglog VS opentelemetry-go

To keep tabs on what’s happening, we use Axiom for logs and Grafana for dashboards and alerts. We pipe all of our logs from most services straight into Axiom and have some pre-defined filters there that help us understand whats going on. With Grafana we have a bunch of custom dashboards that help us triage issues, and more importantly, alert us if some metrics are looking off. Without Grafana, Sliplane could not exist!

I've heard good things about Axiom[0], especially for high scale needs.

0: https://axiom.co/

If your app is instrumented with OpenTelemetry, I'd really recommend something like SigNoz (https://signoz.io/) or Axum (https://axiom.co/) (or even Baselime.io but their application is a mess to work with)

1. Axiom has a generous free plan (not sure if it's self-hostable)

Great choice of monitoring and analytics tools (Sentry, Axiom, Posthog and Uptime Kuma) coupled with amazing Slack integrations that allowed us to iron out any issues way before the traffic spike while the troubling features were still fresh from the oven.

For the last 1.5 years, I have been using Axiom for all of my logs ingestion, querying, and monitoring needs. It is a great product and I never had one issue with it in my time using it. Spoiler alert, even today, when it failed it was actually my fault, but let's see what happened.

Axiom

Axiom is an observability backend with OpenTelemetry support. I chose it for my recent project because of its cost, especially for hobby projects or serious projects in its early phase. They provide a JavaScript library that you can use as transport for pino.

Axiom has a great integration with Next.js: https://axiom.co

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago (https://github.com/open-telemetry/opentelemetry-go/issues/67...).

I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are:

- Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch.

- Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle and tricky.

- Directly marshal protos instead of reflection with a hand-rolled library or with https://github.com/VictoriaMetrics/easyproto.

The gold standard is how TiDB implemented tracing (https://www.pingcap.com/blog/how-we-trace-a-kv-database-with...). Since Go purposefully (and reasonably) doesn't currently provide a comparable abstraction for thread-local storage, we can't implement similar tricks like special-casing when a trace is modified on a single thread.

OpenTelemetry GitHub Repository

When using Hono, you can leverage Fiberplane's client library, which instruments the application based on OpenTelemetry. This allows you to use Fiberplane Studio, which not only displays your endpoints and helps you make requests or replay your webhook but also captures the call chain (traces) among different integrations.

Many development teams leverage OpenTelemetry (OTeL) on the server side to collect signals such as logs, traces, and metrics from their applications. However, what's often overlooked is the power of OTeL's browser instrumentation. This client-side observability data can provide meaningful insights and create a comprehensive view of an application's performance by connecting traces between the client and server.

This is where things get complicated. Implementing such a system is a lot of work, and you should rely on upcoming standards like the W3C recommendation on trace contexts. You should rely on a standard because they solve two major pain points with observability: (1) they have popular implementations like OpenTelemetry (OTel) – which happens to be the second most active CNCF project to this day – and (2) they are widely adopted by major observability backends (e.g., Datadog).

The Vercel AI SDK now supports tracing with OpenTelemetry, an open-source standard for recording telemetry information, as an experimental feature. Here is an example of how trace visualization looks with the Vercel Datadog integration:

I'm not quite sure how to place this. For general app telemetry there's https://opentelemetry.io and for tracking user behavior similar to how it's explained in your docs it looks like https://segment.io would be the closest competitor.

How is telemetry.sh different from the above-mentioned?

Bruno from Inngest here, thanks for asking!

In general, we use OpenTelemetry[1] for instrumenting our services in production, collecting metrics and logs for important events. Specifically, we have set up

- multiple dashboards informing us about current system usage (events received, processed) including e2e latency distributions, compute resource usage for different deployments, and top operations

- metrics on critical systems (data stores including Redis, messaging infrastructure, connection poolers for Postgres, etc.) to gauge current resource utilization and typical load patterns

- alerting on unexpected deviations in KPIs (a subset of the metrics above) to help us spot and react to issues quickly

- forecasting on product usage and compute resource utilization patterns for planning medium to long-term infrastructure work

Hope this helps!

[1]: https://opentelemetry.io/

OpenTelemetry is the future of observability. It is being pushed by Google, Microsoft and Amazon to provide a vendor-neutral solution for traces, metrics and logs. The OpenTelemetry API implementation is available for many programming languages. I will only use metrics and the implementation for the language C# for this tutorial. Let's see how far apart reality and desire are.