hyperdx VS tenzir

Hi HN, Mike and Warren here! We’ve been building an open source local-dev-friendly mode for HyperDX (hyperdx.io). It's a single Docker container that lets you view logs, metrics, and traces for local development in a similar way you can use them for production (live tail, filter/correlate logs & spans, build charts, flamegraph, etc.) Basically, imagine you can run Datadog in a single container for local development. We does this by spinning up an OpenTelemetry collector, Clickhouse DB and HyperDX UI all in a single container - it only takes a few seconds to start and lets you start live tailing your local logs and traces immediately (and graph metrics of course).

README (w/ demo gif): https://github.com/hyperdxio/hyperdx/blob/main/LOCAL.md

We started building local mode as it became one of our most loved [1] community issue after sharing HyperDX. It made sense - as we constantly use HyperDX to debug our own issues locally, and have found it to be a huge productivity boost for things like…

1. Tailing multiple container logs (and grepping/isolating them) without a ton of different terminal splits open at the same time.

2. Be able to sensibly view structured logs and correlated traces to debug local issues (because you’re using structured logging and tracing right?) instead of adding ad-hoc console statements that print out exactly what I could’ve gotten from tracing.

3. Testing telemetry are actually emitting/correlating as expected, before shipping it all the way out to prod and realizing I accidentally created a very expensive high-cardinality metric or a span that has the wrong properties.

We spent some time packaging & tuning our existing OSS stack to run in a single container with less memory/space requirements by staring at `dive` to slim down the image and applied incantations from documentation until memory usage improved. Additionally we removed a few non-local-friendly things like authentication requirements and extraneous services.

It still has all the goodness you’d want - so you can full text search your logs/traces, live tail all your events, view spans correlated with logs (and vice versa), create dashboards based on logs, metrics, traces, and is fully OpenTelemetry compatible - just point your Otel SDK/collector to http://localhost:4318 (or 4317 for the grpc folks) and you’re already good to go.

I’m excited to share what we’ve been working on and would love to hear your feedback and opinions!

Spin up the container yourself to try it out:

docker run -p 8000:8000 -p 4318:4318 -p 4317:4317 -p 8080:8080 -p 8002:8002 hyperdx/hyperdx-local

Main Open Source Repo: https://github.com/hyperdxio/hyperdx

Hosted Demo (in case you want to play around in a cloud sandbox instead): https://api.hyperdx.io/login/demo

HyperDX Landing Page: https://hyperdx.io

[1]: https://github.com/hyperdxio/hyperdx/issues/7

ClickHouse also excels at storing and querying semi-structured data, like event logs. Previously, many engineering teams used Elasticsearch in a similar niche to ClickHouse, building applications like Kibana. Increasingly, developers are choosing ClickHouse over Elasticsearch for its unparalleled performance characteristics. For example, our friends at hyperdx.io are using ClickHouse to build an open-source OpenTelemetry provider!

HyperDX

Biased as a founder in the space [1] but I think with OpenTelemetry + OSS extensible observability tooling, the holy grail of one tool is more realizable than ever.

Vendor lock in with Otel now is hopefully a thing of the past - but now that more obs solutions are going open source, hopefully it's not necessarily true that one tool would be mediocre over all use cases (since DD and the likes are inherently limited by their own engineering teams, vs OSS products can have community/customer contributions to improve the surface area over time on top of the core maintainer's work).

[1] https://github.com/hyperdxio/hyperdx

HyperDX (YC S22) | Founding Engineer | SF Bay Area HQ | REMOTE

We're building an open source, dev friendly observability tool (think Datadog, but something developers actually love to use and companies can actually afford).

We're in the intersection of needing to build rock solid infrastructure ingesting TBs of data, searching it incredibly quickly and scalably, and layering on top a buttery smooth DX from our language-specific SDKs, APIs and web app.

We're super early and hiring our first founding engineer. We already have a cloud product customers pay for and love, loads of runway regardless of the wider economy, 5k+ Github stars weeks after our OSS launch, and tons of hard technical problems.

The vast majority of our work is open source, so you can get a sense of what you'd be working with here: https://github.com/hyperdxio/hyperdx

Our job listing is here as well: https://www.ycombinator.com/companies/hyperdx/jobs/zFXTbzl-f...

Come by our discord as well if you just want to talk shop: https://discord.gg/FErRRKU78j

I'm Mike, one of the cofounders. If you love shipping quickly and want to help us build from the ground up an open source developer tool (that devs won't hate when they're on-call), let me know! mike [at] hyperdx.io

Oh huh! What would you do with span start independent of stops?

A tangent on logcat - local observability to me is a really intriguing area, I think there's a story of Otel for local as well if someone can build a good enough local DX for consuming them (we've been told a number of times about this https://github.com/hyperdxio/hyperdx/issues/7 as an example)

Hi HN, Mike and Warren here! We’ve been building HyperDX (hyperdx.io). HyperDX allows you to easily search and correlate logs, traces, metrics (alpha), and session replays all in one place. For example, if a user reports a bug “this button doesn’t work," an engineer can play back what the user was doing in their browser and trace API calls back to the backend logs for that specific request, all from a single view. Github Repo: https://github.com/hyperdxio/hyperdx

We're building the OSS equivalent when it comes to the observability side of Splunk/DD, on Clickhouse naturally of course but believe in the same end goal of lowering cost via separation of compute and storage.

https://github.com/hyperdxio/hyperdx

We're building something similar at Tenzir, but more for operational security workloads. https://docs.tenzir.com

Differences to Vector:

- An agent has optional indexed storage, so you can store your data there and pick it up later. The storage is based on Apache Feather, Parquet's little brother.

- Pipelines operators both work with data frames (Arrow record batches) or chunks of bytes.

- Structured pipelines are multi-schema, i.e., a single pipeline can process streams of record batches with different schemas.

Tenzir | Remote (EU) or Hamburg, Germany | open-core | Full-time | https://tenzir.com

Tenzir is hiring several key engineering roles to meet the needs in expanding the team. Our product: security data pipelines. From the data side, think of it as an Arrow-native, multi-schema ETL tool that offers optional storage in Parquet/Feather. From the security perspective, think of it as a solution for collecting, parsing, transforming, aggregating, and routing data. We typically sit between the data sources (endpoint, network, cloud) and sinks (SIEM, data lake).

Our open-source execution engine is C++20 (https://github.com/tenzir/tenzir), our platform is SvelteKit and TypeScript. Experience with data-first frontend apps is a great plus. Open positions at https://tenzir.jobs.personio.de:

    - Fullstack Engineer

We're in the middle of getting TQL v2 [] out of the door with support for expressions and more advanced control flow, e.g., match-case statements. There's a blog post [#] about the core design of the engine as well.

While it's a general-purpose ETL tool, we're targeting primary operational security use case where people today use Splunk, Sentinel/ADX, Elastic, etc. So some operators are very security'ish, like Sigma, YARA, or Velociraptor.

[] https://github.com/tenzir/tenzir/blob/64ef997d736e9416e859bf...

[#] https://docs.tenzir.com/blog/five-design-principles-for-buil...

Hey, founder of Tenzir [1] here — We are building an open-core pipeline-first security data engine that can massively reduce your Splunk costs. Even though we go to market "mid stream" we have a few users that use us as light-weight SIEM (or more accurately, just plain log management).

We are still in early access to browse through our docs or swing by our Discord.

[1] https://tenzir.com | https://docs.tenzir.com

Download VAST v3.1 here: https://github.com/tenzir/vast/releases/tag/v3.1.0

Tenzir is a funded seed-stage startup that builds a next generation data-plane for plug-and-play security operations. Our mission is to empower defenders with an open data engineering platform to perform data-driven investigations through combination best-of-breed solutions. Our stack consists of the high-performance C++20 telemetry engine VAST, a Rust API, and a ReasonML-based frontend.

I had chat with Hannes, the DuckDB co-founder, a few weeks ago. They are building awesome stuff to become the "SQLite of OLAP". The team comes with a strong academic background and is tuned into the data engineering world.

At Tenzir, we looked at DuckDB as embeddable backend engine to do the heavy lifting of query execution of our engine [1]. Our idea is throwing over a set of Parquet files, along with a query; initially SQL but perhaps soon Substrait [2] if it picks up.

We also experiment with a cloud deployment [3] where a different set of I/O path may warrant a different backend engine. Right now, we're working on a serverless approach leveraging Datafusion (and depending on maturity, Ballista at some point).

My hunch is that we will see more pluggability in this space moving forward. It's not only meaningful from an open-core business model perspective, but also pays dividends to the UX. The company that's solving a domain problem (for us: security operations center infrastructre) can leverage a high-bandwidth drop-in engine and only needs to wire it properly. This requires much less data engineers than building a poorman's version of the same inhouse.

We also have the R use case, e.g., to write reports in Rmarkdown that crunch some customer security telemetry, highlighting outliers or other noteworthy events. We're not there yet, but with the right query backend, I would expect to get this almost for free. We're close to being ready to use Arrow Flight for interop, but it's not zero-copy. DuckDB has demonstrated the zero-copy approach recently [4], going through the C API. (The story is also relevant when doing s/R/Python/, FWIW.)

[1] https://github.com/tenzir/vast

To this end, we build the high-performance telemetry engine VAST, which at its core, ingests hundreds of thousands of events per second from high-volume data sources (such as network telemetry as NetFlow, Zeek, Suricata, and endpoint telemetry from various agents). To the user, VAST offers low-latency access through various APIs, and in particular Apache Arrow for high-bandwidth data sharing with downstream tooling. A flexible plugin API enables additional security-specific use cases on top, such as realtime matching of threat intelligence or mining of asset data for passive inventorization.

Tenzir | C++, ReasonML, Rust, Python | Remote | Open-source | Full-time | https://tenzir.com

Tenzir is a funded seed-stage startup that builds a next generation data-plane for plug-and-play security operations. Our mission is to empower defenders with an open platform to perform automated data-driven investigations through combination best-of-breed solutions. Our stack consists of the high-performance C++ database VAST (https://github.com/tenzir/vast), a Rust API, and a ReasonML-based frontend.

Our open engineering positions include:

- Database: https://tenzir.com/career/backend-engineer/

- DevOps: https://tenzir.com/career/devops-platform-engineer/

- Frontend: https://tenzir.com/career/frontend-engineer/

We are based out of Hamburg, Germany, but cultivate an agile remote-first mindset. If you live in the region and look for a System Administrator, we’d love to hear from you!

For any questions, feel free to reach out to us at [email protected].

We at Tenzir (https://tenzir.com/) are an early-stage startup that build a next generation data-plane for modern Security Operations Centers. We are looking for a frontend engineer to help us enhance the web interface to VAST (our open-core telemetry engine, https://github.com/tenzir/vast). In our stack, we use C++ for VAST , Rust and ReasonML (compiled to JS) in our API-Layer, and ReasonML on the frontend. Our website is written in ReasonML with the help of Gatsby. Our team cultivates a mindset of strong typing and functional programming, practiced end-to-end across the entire stack. We're a remote-first company, scattered across Europe. Ideally looking for someone within (+ / -) 4hrs timezone.