hyperloglog VS fluent-bit

To keep tabs on what’s happening, we use Axiom for logs and Grafana for dashboards and alerts. We pipe all of our logs from most services straight into Axiom and have some pre-defined filters there that help us understand whats going on. With Grafana we have a bunch of custom dashboards that help us triage issues, and more importantly, alert us if some metrics are looking off. Without Grafana, Sliplane could not exist!

I've heard good things about Axiom[0], especially for high scale needs.

0: https://axiom.co/

If your app is instrumented with OpenTelemetry, I'd really recommend something like SigNoz (https://signoz.io/) or Axum (https://axiom.co/) (or even Baselime.io but their application is a mess to work with)

1. Axiom has a generous free plan (not sure if it's self-hostable)

Great choice of monitoring and analytics tools (Sentry, Axiom, Posthog and Uptime Kuma) coupled with amazing Slack integrations that allowed us to iron out any issues way before the traffic spike while the troubling features were still fresh from the oven.

For the last 1.5 years, I have been using Axiom for all of my logs ingestion, querying, and monitoring needs. It is a great product and I never had one issue with it in my time using it. Spoiler alert, even today, when it failed it was actually my fault, but let's see what happened.

Axiom

Axiom is an observability backend with OpenTelemetry support. I chose it for my recent project because of its cost, especially for hobby projects or serious projects in its early phase. They provide a JavaScript library that you can use as transport for pino.

Axiom has a great integration with Next.js: https://axiom.co

I kept an eye on both Fluent Bit and Vector for a while. I decided to pick one to test in production, in particular to:

Second, download the parsers.conf from the official repo, mount it into the Fluent Bit container, and change the startup script to this:

git clone https://github.com/fluent/fluent-bit.git cd fluent-bit cmake . make sudo make install

In this lab, you will use *Fluent Bit *to route logs. The lab order will install CloudWatch Agent **to collect metric of the cluster and **Fluent Bit to send logs to CloudWatch Logs in DaemonSet type.

In this section, we'll dive into creating and deploying a data processing platform on Amazon Elastic Kubernetes Service Amazon EKS.   The solution includes essential Kubernetes add-ons: Argo Workflows, Argo Events, Spark Operator for managing Spark jobs, Fluent Bit for logging, and Prometheus for metrics.

We’ve shown one of the almost endless approaches to monitoring IRIS applications deployed in GKE. This time we’ve focused on metrics stored in Cloud Monitoring and displayed in Grafana. But don’t forget about IRIS logs. Pods logs are, at the moment of writing, gathered by fluentbit and sent to Cloud Logging where they can be viewed.

Some of the HN discussion about whether "new projects in C should be allowed" is moot: Fluent Bit was imported into git in 2015 [0] (a few months before Rust's first public release), and may be considerably older than that for all I know.

I suppose incidents like this actually do give a reason to "rewrite it in Rust", when "it" is "widely deployed infrastructure written in C". OTOH, I'm sure there were plenty of non-memory-safety bugs introduced and later fixed over the years, and rewriting in Rust will recapitulate that subset of bugs.

[0] https://github.com/fluent/fluent-bit/commit/49269c5ec3c74411...

Fluentbit

$ docker run --rm fluent-bit-dummy WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/12/24 16:06:59] [ info] [fluent bit] version=1.9.10, commit=557c8336e7, pid=1 [2023/12/24 16:06:59] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2023/12/24 16:06:59] [ info] [cmetrics] version=0.3.7 [2023/12/24 16:06:59] [ info] [output:stdout:stdout.0] worker #0 started [2023/12/24 16:06:59] [ info] [sp] stream processor started [0] dummy.0: [1703434019.553880465, {"message"=>"custom dummy"}] [0] dummy.0: [1703434020.555768799, {"message"=>"custom dummy"}] [0] dummy.0: [1703434021.550525174, {"message"=>"custom dummy"}] [0] dummy.0: [1703434022.551563050, {"message"=>"custom dummy"}] [0] dummy.0: [1703434023.551944509, {"message"=>"custom dummy"}] [0] dummy.0: [1703434024.550027843, {"message"=>"custom dummy"}] [0] dummy.0: [1703434025.550901801, {"message"=>"custom dummy"}] [0] dummy.0: [1703434026.549279385, {"message"=>"custom dummy"}] ^C[2023/12/24 16:07:08] [engine] caught signal (SIGINT) [0] dummy.0: [1703434027.549678344, {"message"=>"custom dummy"}] [2023/12/24 16:07:08] [ warn] [engine] service will shutdown in max 5 seconds [2023/12/24 16:07:08] [ info] [engine] service has stopped (0 pending tasks) [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopping... [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopped