opentelemetry-go VS hyperdx

Our first approach was to implement a separate SDK for each independent technology stack. We decided to use OpenTelemetry which is widely adopted and covers most of our needs.

Distributed system administrators need mechanisms and tools for monitoring individual nodes in order to analyze the system and promptly detect anomalies. Developers also need effective mechanisms for analyzing, diagnosing issues, and identifying bugs in protocol implementations. Logging, tracing, and collecting metrics are common observability techniques to allow monitoring and obtaining diagnostic information from the system; most of the explored code bases use these techniques. OpenTelemetry and Prometheus are popular open-source monitoring solutions, which are used in many of the explored code bases.

OpenTelemetry

When choosing distributed tracing tools, considerations include your technology stack, business requirements, and monitoring complexity. Zipkin, SkyWalking, and OpenTelemetry are popular distributed tracing solutions, each with its unique features.

You can follow this process with any large token AI system like Claude by identifying tracing data relevant to the code you are working on, using it as context to prompt OpenAI or other LLMs. Generally, you’d generate tracing data by implementing OpenTelemetry (aka OTEL) libraries into your application, adding spans to your functions with Jaeger, or using commercial SaaS tools like Honeycomb and Datadog.

While many programming languages provide robust support for Open Telemetry, this instance focuses on Golang. It's important to note that, in the current context, the logs SDK for Golang is not implemented. For future reference consult the list of supported languages and explore the Open Telemetry repositories. Always prioritize the main repository and its contrib repository, housing extensions and instrumentation libraries crucial to the Open Telemetry framework. Stay updated with the latest developments to ensure seamless integration and enhanced functionality.

OneUptime (https://github.com/oneuptime/oneuptime) is the open-source alternative to DataDog. It's 100% free and you can self-host it on your VM / server / cloud or you can use SaaS at https://oneuptime.com

NEW UPDATES (since we last posted to HN): We now support OpenTelemetry (https://opentelemetry.io/) natively which will help you to monitor, observe and debug any app, service, database or stack.

OpenTelemetry is the fastest growing Cloud Native Computing Foundation (CNCF) project. It standardizes the instrumentation and collection of traces, metrics, and logs from applications, and is supported by all the major observability projects, languages, and tools. One standard to rule them all!

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. https://opentelemetry.io/

Tracetest uses your existing OpenTelemetry traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source, or send traces to Tracetest directly!

Hi HN, Mike and Warren here! We’ve been building an open source local-dev-friendly mode for HyperDX (hyperdx.io). It's a single Docker container that lets you view logs, metrics, and traces for local development in a similar way you can use them for production (live tail, filter/correlate logs & spans, build charts, flamegraph, etc.) Basically, imagine you can run Datadog in a single container for local development. We does this by spinning up an OpenTelemetry collector, Clickhouse DB and HyperDX UI all in a single container - it only takes a few seconds to start and lets you start live tailing your local logs and traces immediately (and graph metrics of course).

README (w/ demo gif): https://github.com/hyperdxio/hyperdx/blob/main/LOCAL.md

We started building local mode as it became one of our most loved [1] community issue after sharing HyperDX. It made sense - as we constantly use HyperDX to debug our own issues locally, and have found it to be a huge productivity boost for things like…

1. Tailing multiple container logs (and grepping/isolating them) without a ton of different terminal splits open at the same time.

2. Be able to sensibly view structured logs and correlated traces to debug local issues (because you’re using structured logging and tracing right?) instead of adding ad-hoc console statements that print out exactly what I could’ve gotten from tracing.

3. Testing telemetry are actually emitting/correlating as expected, before shipping it all the way out to prod and realizing I accidentally created a very expensive high-cardinality metric or a span that has the wrong properties.

We spent some time packaging & tuning our existing OSS stack to run in a single container with less memory/space requirements by staring at `dive` to slim down the image and applied incantations from documentation until memory usage improved. Additionally we removed a few non-local-friendly things like authentication requirements and extraneous services.

It still has all the goodness you’d want - so you can full text search your logs/traces, live tail all your events, view spans correlated with logs (and vice versa), create dashboards based on logs, metrics, traces, and is fully OpenTelemetry compatible - just point your Otel SDK/collector to http://localhost:4318 (or 4317 for the grpc folks) and you’re already good to go.

I’m excited to share what we’ve been working on and would love to hear your feedback and opinions!

Spin up the container yourself to try it out:

docker run -p 8000:8000 -p 4318:4318 -p 4317:4317 -p 8080:8080 -p 8002:8002 hyperdx/hyperdx-local

Main Open Source Repo: https://github.com/hyperdxio/hyperdx

Hosted Demo (in case you want to play around in a cloud sandbox instead): https://api.hyperdx.io/login/demo

HyperDX Landing Page: https://hyperdx.io

[1]: https://github.com/hyperdxio/hyperdx/issues/7

ClickHouse also excels at storing and querying semi-structured data, like event logs. Previously, many engineering teams used Elasticsearch in a similar niche to ClickHouse, building applications like Kibana. Increasingly, developers are choosing ClickHouse over Elasticsearch for its unparalleled performance characteristics. For example, our friends at hyperdx.io are using ClickHouse to build an open-source OpenTelemetry provider!

HyperDX

Biased as a founder in the space [1] but I think with OpenTelemetry + OSS extensible observability tooling, the holy grail of one tool is more realizable than ever.

Vendor lock in with Otel now is hopefully a thing of the past - but now that more obs solutions are going open source, hopefully it's not necessarily true that one tool would be mediocre over all use cases (since DD and the likes are inherently limited by their own engineering teams, vs OSS products can have community/customer contributions to improve the surface area over time on top of the core maintainer's work).

[1] https://github.com/hyperdxio/hyperdx

HyperDX (YC S22) | Founding Engineer | SF Bay Area HQ | REMOTE

We're building an open source, dev friendly observability tool (think Datadog, but something developers actually love to use and companies can actually afford).

We're in the intersection of needing to build rock solid infrastructure ingesting TBs of data, searching it incredibly quickly and scalably, and layering on top a buttery smooth DX from our language-specific SDKs, APIs and web app.

We're super early and hiring our first founding engineer. We already have a cloud product customers pay for and love, loads of runway regardless of the wider economy, 5k+ Github stars weeks after our OSS launch, and tons of hard technical problems.

The vast majority of our work is open source, so you can get a sense of what you'd be working with here: https://github.com/hyperdxio/hyperdx

Our job listing is here as well: https://www.ycombinator.com/companies/hyperdx/jobs/zFXTbzl-f...

Come by our discord as well if you just want to talk shop: https://discord.gg/FErRRKU78j

I'm Mike, one of the cofounders. If you love shipping quickly and want to help us build from the ground up an open source developer tool (that devs won't hate when they're on-call), let me know! mike [at] hyperdx.io

Oh huh! What would you do with span start independent of stops?

A tangent on logcat - local observability to me is a really intriguing area, I think there's a story of Otel for local as well if someone can build a good enough local DX for consuming them (we've been told a number of times about this https://github.com/hyperdxio/hyperdx/issues/7 as an example)

Hi HN, Mike and Warren here! We’ve been building HyperDX (hyperdx.io). HyperDX allows you to easily search and correlate logs, traces, metrics (alpha), and session replays all in one place. For example, if a user reports a bug “this button doesn’t work," an engineer can play back what the user was doing in their browser and trace API calls back to the backend logs for that specific request, all from a single view. Github Repo: https://github.com/hyperdxio/hyperdx

We're building the OSS equivalent when it comes to the observability side of Splunk/DD, on Clickhouse naturally of course but believe in the same end goal of lowering cost via separation of compute and storage.

https://github.com/hyperdxio/hyperdx