logs-benchmark VS clamshell

I am a maintainer at SigNoz. Nice to see OpenObserve's belief that the future of Observability should be OpenSource. We chose clickhouse rather than building a database as it takes multi-year effort to move a db to maturity and clickhouse has been battle-tested at Yandex, Uber and Cloudflare. Clickhouse also provides native integration with s3 and other blob storages. Our users have been using disk as hot storage for a week and moving the data to s3 after that. Tiered storage is really cool in terms of query performance.

We have also seen logs data of our users at a compression ratio of 30x/40x. We have published a logs benchmark (https://github.com/SigNoz/logs-benchmark) where the data is very high-cardinal (causing a compression factor of only 2.5x). Would love to see how does OpenObserve perform in that dataset someday.

Wishing you best for the journey ahead.

Did you update the benchmark after you got feedback from this user at Hacker News or after you got the feedback that including Loki here is kinda pointless?

This is well made case - but I'm not sure I buy the central argument. Within some basic limits, I don't think terseness and readability have the contradiction made out here, because in programming we have abstraction, which gives us both.

To take the example command that's given:

beef.txt | grep "lasagna" | sort -n | uniq

Sure, writing the logic out for this in something like python straight out the bat with only the standard library might look messy, but with one basic convenience function it could quickly be:

search(for='lasagna', in='beef.txt', clear_duplicates=False).sorted()

Obviously you have to write the function in the first place, but I'd say if you're doing something like this often, it's easily worth spending that 2 minutes. And if you're not doing this often, you'll have a faster time writing more code, but keeping less heavy lifting of "how does bash pipe together" in your head.

I shared a project here a few weeks ago experimenting with what my dream shell might look like, what surprised me more than anything else, was how easy writing a repl environment actually is. I put a scrappy one together as one person in a few hours, so I don't understand why as developers we've reached general language models before being able to make a powerful, but new-user friendly shell.

Also, completely unrelated note, but posix only allows passing back strings - but isn't this true of web apis too which we use all the time? How come no json as a standard passback from programs?

Shameless plug for the project I mentioned earlier: https://github.com/benrutter/clamshell

clamshell – experimenting with a python based shell

Check it out here!