I Don't Think Elasticsearch Is a Good Logging System

• ClickHouse

  218 36,676 10.0 C++

  ClickHouse® is a real-time analytics DBMS

  

• InfluxDB

  www.influxdata.com featured

  Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

  

• elasticsearch-py

  21 4,213 8.9 Python

  Official Python client for Elasticsearch

  

  Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

• syslog-ng

  8 2,107 9.9 C

  syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

  

  What we do internally is use [syslog-ng](https://github.com/syslog-ng/syslog-ng) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

• vector

  101 17,455 9.9 Rust

  A high-performance observability data pipeline.

  

  Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes.

  I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.

  I can't let a ELK discussion pass without mentioning vector.dev (https://vector.dev) which I'm not affiliated with aside from being a very happy user (for log ingestion).

• qryn

  10 1,165 9.4 JavaScript

  qryn is a polyglot, high-performance observability framework for ClickHouse. Ingest, store and analyze logs, metrics and telemetry traces from any agent supporting Loki, Prometheus, OTLP, Tempo, Elastic, InfluxDB and many more formats and query transparently using Grafana or any other compatible client.

  

  There's also cLoki. It's a new project that puts a Loki gateway over a ClickHouse backend store. We're looking at it and plan a presentation from the author(s) at the next ClickHouse SF Bay Area Meetup.

  https://github.com/lmangani/cLoki

• zeek-clickhouse

  1 12 0.0 Go

  > Uber has not open sourced this work so we are unable to benchmark it and see how it performs

  I implemented their design here, specifically for importing zeek logs:

  https://github.com/JustinAzoff/zeek-clickhouse

  I don't have the elastic compatible query api though, or the smarts that auto materialize popular columns.

  It works though, does a good job at soaking up any sort of log type and handling fields being added or removed.

• helm-charts

  20 1,599 9.7 Smarty

  

  Out of Order support is available in Loki's main branch and included in the next release. It's already live in production on Grafana Cloud. https://grafana.com/blog/2021/09/16/avoid-dropped-logs-due-t...

  High availability in Loki is supported in distributed mode. Helm chart here: https://github.com/grafana/helm-charts/tree/main/charts/loki...

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• systemd

  545 13,038 10.0 C

  The systemd System and Service Manager

  

  Yeah, that's exactly why I wrote my comment. :)

  It should be able to do that, but not really. And it seems easy to fix, but of course patches are welcome. (Hopefully.)

  https://github.com/systemd/systemd/issues/5242

Suggest a related project