nano VS quickwit

Some time ago I adopted Quickwit at my company. For anyone who hasn't used it: Quickwit is a search engine that runs full-text search directly on object storage (S3 or anything S3-compatible). It decouples compute from storage, so you don't pay to keep big indexes warm to search older data. That model fits logs well.

Nice... it's cool to see how different companies are putting together best fit solutions. I'm also glad that they at least started out with off the shelf apps instead of jumping to something like a bespoke solution early on.

Quickwit[1] looks interesting, found via Tantivity reference. Kind of like ES w/ Lucene.

1. https://github.com/quickwit-oss/quickwit

- https://github.com/quickwit-oss/quickwit

No matter how you twist and turn, how many initialization configs you disable, OpenSearch/ES is a beast, heavy, takes long to start and eats up a lot of RAM, and I really really wouldn’t try to ship this without docker. So how can we make the pip install dream work? Luckily, we found a nice little thing called quickwit. Quickwit is built for scale, but its rust-compiled binary can be downloaded for just ~80mb, it starts very fast AND it has an elasticsearch-somewhat-compatible API! Just what we need. Now it doesn’t support scripts or some complicated aggregations, but that’s fine, you don’t need it all on dev mode, and this is where we’re at right now.

Can you clarify: Does the full-text search for logs linearly search all logs like Loki does, or can it speed it up with an index?

The docs at https://www.hyperdx.io/docs/search don't seem to talk about this key design decision.

I have a couple 100 GB to few TB logs (all from `journald` or JSON lines), just want to store them forever, and find results fast when searching for arbitrary substrings.

Loki does not use an index, so it's pretty slow at finding results in TB-sized logs (does not return results within a few seconds, so it's not interactive).

https://quickwit.io is one thing I'm looking at integrating, that can solve much of the index-based log search.

(Note I'm not super familar with the capabilities of ClickHouse itself regarding indexed full-text search.)

If you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, and it's OSS

Repo: https://github.com/quickwit-oss/quickwit

Luckily, viable open-source alternatives like Quickwit are here to come to the rescue. By weaving together existing tooling for log and trace ingesting as well as pairing well with dashboard and visualisation tools such as Grafana and Jaeger. And sandwiching powerful indexing storage and search capabilities in between. Even if the tool sounds new, it won’t be for long.

Quickwit is a cloud-native search engine that emerged with the goal of creating an open-source alternative to expensive monitoring software like Datadog/Splunk. With its robust Elasticsearch-compatible API, Quickwit integrates well with tooling from the OSS ecosystem, such as Grafana, Jaeger, and OpenTelemetry.

We have partnered with third party packages such as Keptn and Quickwit to expedite their integration onto Glasskube.