I Don't Think Elasticsearch Is a Good Logging System

• ClickHouse

  208 34,269 10.0 C++

  ClickHouse® is a free analytics DBMS for big data

  

• elasticsearch-py

  21 4,143 8.9 Python

  Official Python client for Elasticsearch

  

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• syslog-ng

  8 2,035 9.9 C

  syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.

  

What we do internally is use [syslog-ng](https://github.com/syslog-ng/syslog-ng) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

• vector

  96 16,561 9.9 Rust

  A high-performance observability data pipeline.

  

Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes.

I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.

I can't let a ELK discussion pass without mentioning vector.dev (https://vector.dev) which I'm not affiliated with aside from being a very happy user (for log ingestion).

• qryn

  10 947 9.6 JavaScript

  qryn is a polyglot, high-performance observability framework for ClickHouse. Ingest, store and analyze logs, metrics and telemetry traces from any agent supporting Loki, Prometheus, OTLP, Tempo, Elastic, InfluxDB and many more formats and query transparently using Grafana or any other compatible client.

  

There's also cLoki. It's a new project that puts a Loki gateway over a ClickHouse backend store. We're looking at it and plan a presentation from the author(s) at the next ClickHouse SF Bay Area Meetup.

https://github.com/lmangani/cLoki

• zeek-clickhouse

  1 11 0.0 Go

> Uber has not open sourced this work so we are unable to benchmark it and see how it performs

I implemented their design here, specifically for importing zeek logs:

https://github.com/JustinAzoff/zeek-clickhouse

I don't have the elastic compatible query api though, or the smarts that auto materialize popular columns.

It works though, does a good job at soaking up any sort of log type and handling fields being added or removed.

• helm-charts

  19 1,497 9.7 Smarty
  

Out of Order support is available in Loki's main branch and included in the next release. It's already live in production on Grafana Cloud. https://grafana.com/blog/2021/09/16/avoid-dropped-logs-due-t...

High availability in Loki is supported in distributed mode. Helm chart here: https://github.com/grafana/helm-charts/tree/main/charts/loki...

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• systemd

  518 12,516 10.0 C

  The systemd System and Service Manager

  

Yeah, that's exactly why I wrote my comment. :)

It should be able to do that, but not really. And it seems easy to fix, but of course patches are welcome. (Hopefully.)

https://github.com/systemd/systemd/issues/5242

Suggest a related project