hyperdx VS plumber

Hi HN, Mike and Warren here! We’ve been building an open source local-dev-friendly mode for HyperDX (hyperdx.io). It's a single Docker container that lets you view logs, metrics, and traces for local development in a similar way you can use them for production (live tail, filter/correlate logs & spans, build charts, flamegraph, etc.) Basically, imagine you can run Datadog in a single container for local development. We does this by spinning up an OpenTelemetry collector, Clickhouse DB and HyperDX UI all in a single container - it only takes a few seconds to start and lets you start live tailing your local logs and traces immediately (and graph metrics of course).

README (w/ demo gif): https://github.com/hyperdxio/hyperdx/blob/main/LOCAL.md

We started building local mode as it became one of our most loved [1] community issue after sharing HyperDX. It made sense - as we constantly use HyperDX to debug our own issues locally, and have found it to be a huge productivity boost for things like…

1. Tailing multiple container logs (and grepping/isolating them) without a ton of different terminal splits open at the same time.

2. Be able to sensibly view structured logs and correlated traces to debug local issues (because you’re using structured logging and tracing right?) instead of adding ad-hoc console statements that print out exactly what I could’ve gotten from tracing.

3. Testing telemetry are actually emitting/correlating as expected, before shipping it all the way out to prod and realizing I accidentally created a very expensive high-cardinality metric or a span that has the wrong properties.

We spent some time packaging & tuning our existing OSS stack to run in a single container with less memory/space requirements by staring at `dive` to slim down the image and applied incantations from documentation until memory usage improved. Additionally we removed a few non-local-friendly things like authentication requirements and extraneous services.

It still has all the goodness you’d want - so you can full text search your logs/traces, live tail all your events, view spans correlated with logs (and vice versa), create dashboards based on logs, metrics, traces, and is fully OpenTelemetry compatible - just point your Otel SDK/collector to http://localhost:4318 (or 4317 for the grpc folks) and you’re already good to go.

I’m excited to share what we’ve been working on and would love to hear your feedback and opinions!

Spin up the container yourself to try it out:

docker run -p 8000:8000 -p 4318:4318 -p 4317:4317 -p 8080:8080 -p 8002:8002 hyperdx/hyperdx-local

Main Open Source Repo: https://github.com/hyperdxio/hyperdx

Hosted Demo (in case you want to play around in a cloud sandbox instead): https://api.hyperdx.io/login/demo

HyperDX Landing Page: https://hyperdx.io

[1]: https://github.com/hyperdxio/hyperdx/issues/7

ClickHouse also excels at storing and querying semi-structured data, like event logs. Previously, many engineering teams used Elasticsearch in a similar niche to ClickHouse, building applications like Kibana. Increasingly, developers are choosing ClickHouse over Elasticsearch for its unparalleled performance characteristics. For example, our friends at hyperdx.io are using ClickHouse to build an open-source OpenTelemetry provider!

HyperDX

Biased as a founder in the space [1] but I think with OpenTelemetry + OSS extensible observability tooling, the holy grail of one tool is more realizable than ever.

Vendor lock in with Otel now is hopefully a thing of the past - but now that more obs solutions are going open source, hopefully it's not necessarily true that one tool would be mediocre over all use cases (since DD and the likes are inherently limited by their own engineering teams, vs OSS products can have community/customer contributions to improve the surface area over time on top of the core maintainer's work).

[1] https://github.com/hyperdxio/hyperdx

HyperDX (YC S22) | Founding Engineer | SF Bay Area HQ | REMOTE

We're building an open source, dev friendly observability tool (think Datadog, but something developers actually love to use and companies can actually afford).

We're in the intersection of needing to build rock solid infrastructure ingesting TBs of data, searching it incredibly quickly and scalably, and layering on top a buttery smooth DX from our language-specific SDKs, APIs and web app.

We're super early and hiring our first founding engineer. We already have a cloud product customers pay for and love, loads of runway regardless of the wider economy, 5k+ Github stars weeks after our OSS launch, and tons of hard technical problems.

The vast majority of our work is open source, so you can get a sense of what you'd be working with here: https://github.com/hyperdxio/hyperdx

Our job listing is here as well: https://www.ycombinator.com/companies/hyperdx/jobs/zFXTbzl-f...

Come by our discord as well if you just want to talk shop: https://discord.gg/FErRRKU78j

I'm Mike, one of the cofounders. If you love shipping quickly and want to help us build from the ground up an open source developer tool (that devs won't hate when they're on-call), let me know! mike [at] hyperdx.io

Oh huh! What would you do with span start independent of stops?

A tangent on logcat - local observability to me is a really intriguing area, I think there's a story of Otel for local as well if someone can build a good enough local DX for consuming them (we've been told a number of times about this https://github.com/hyperdxio/hyperdx/issues/7 as an example)

Hi HN, Mike and Warren here! We’ve been building HyperDX (hyperdx.io). HyperDX allows you to easily search and correlate logs, traces, metrics (alpha), and session replays all in one place. For example, if a user reports a bug “this button doesn’t work," an engineer can play back what the user was doing in their browser and trace API calls back to the backend logs for that specific request, all from a single view. Github Repo: https://github.com/hyperdxio/hyperdx

We're building the OSS equivalent when it comes to the observability side of Splunk/DD, on Clickhouse naturally of course but believe in the same end goal of lowering cost via separation of compute and storage.

https://github.com/hyperdxio/hyperdx

Streamdal

4. Go to the provided UI (or run the CLI app) and be able to peek into what your app is reading or writing, like with `tail -f`.

And that's basically it. There's a bunch more functionality in the project but we find this to be the most immediately useful part. Every developer we've shown this to has said "I wish I had this at my gig at $company" - and we feel exactly the same. We are devs and this is what we’ve always wanted, hundreds of times - a way to just quickly look at the data our software is producing in real-time, without having to jump through any hoops.

If you want to learn more about the "why" and the origin of this project - you can read about it here: https://streamdal.com/manifesto

— — —

HOW DOES IT WORK?

The SDK establishes a long-running session with the server (using gRPC) and "listens" for commands that are forwarded to it all the way from the UI -> server -> SDK.

The commands are things like: "show me the data that you are currently consuming", "apply these rules to all data that you produce", "inspect the schema for all data", and so on.

The SDK interprets the command and either executes Wasm-based rules against the data it's processing or if it's a `tail` request - it'll send the data to the server, which will forward it to the UI for display.

The SDK IS part of the critical path but it does not have a dependency on the server. If the server is gone, you won't be able to use the UI or send commands to the SDKs, but that's about it - the SDKs will continue to work and attempt to reconnect to the server behind the scenes.

— — —

TECHNICAL BITS

The project consists of a lot of "buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm, Deno, ReactFlow, and probably a few other things.

The server is written in Go, all of the Wasm is Rust and the UI is Typescript. There are SDKs for Go, Python, and Node. We chose these languages for the SDKs because we've been working in them daily for the past 10+ years.

The reasons for the tech choices are explained in detail here: https://docs.streamdal.com/en/resources-support/open-source/

— — —

LAST PART

OK, that's it. What do you think? Is it useful? Can we answer anything?

- If you like what you're seeing, give our repo a star: https://github.com/streamdal/streamdal

Checkout plumber https://github.com/streamdal/plumber if you want all the Postgres changes sent to basically any type of broker queue https://docs.streamdal.com/en/data-ingestion/relay/postgresql-cdc/. I would say Nat's Jetstream is probably the way to go if you have K8s running already. It's a dead simple service written in Go. Just make sure you allocate enough memory to Jetstream.

I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.

For CDC and event driven part of the stack, Plumber is a great free tool. That project is going to be adding sampling soon too - this can def help with the cost of ETL.

https://github.com/batchcorp/plumber check it out if you want to get into event driven systems

That’s awesome to see. My coworker and I always figured Go would be perfect for this. Going to be a serious amount of work! I see you use NATS as well. Big fan of it. Checkout our project https://github.com/batchcorp/plumber if you end up needing to inspect or send messages while deving against it.

Checkout plumber https://github.com/batchcorp/plumber join our slack https://launchpass.com/streamdal we got a pretty knowledgeable group