fluent-bit VS stanza

Fluentbit

$ docker run --rm fluent-bit-dummy WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/12/24 16:06:59] [ info] [fluent bit] version=1.9.10, commit=557c8336e7, pid=1 [2023/12/24 16:06:59] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2023/12/24 16:06:59] [ info] [cmetrics] version=0.3.7 [2023/12/24 16:06:59] [ info] [output:stdout:stdout.0] worker #0 started [2023/12/24 16:06:59] [ info] [sp] stream processor started [0] dummy.0: [1703434019.553880465, {"message"=>"custom dummy"}] [0] dummy.0: [1703434020.555768799, {"message"=>"custom dummy"}] [0] dummy.0: [1703434021.550525174, {"message"=>"custom dummy"}] [0] dummy.0: [1703434022.551563050, {"message"=>"custom dummy"}] [0] dummy.0: [1703434023.551944509, {"message"=>"custom dummy"}] [0] dummy.0: [1703434024.550027843, {"message"=>"custom dummy"}] [0] dummy.0: [1703434025.550901801, {"message"=>"custom dummy"}] [0] dummy.0: [1703434026.549279385, {"message"=>"custom dummy"}] ^C[2023/12/24 16:07:08] [engine] caught signal (SIGINT) [0] dummy.0: [1703434027.549678344, {"message"=>"custom dummy"}] [2023/12/24 16:07:08] [ warn] [engine] service will shutdown in max 5 seconds [2023/12/24 16:07:08] [ info] [engine] service has stopped (0 pending tasks) [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopping... [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopped

> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers.

And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory.

I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling.

[1] https://github.com/fluent/fluent-bit/issues/4836

Fluentbit is Fluentd's lightweight alternative to itself.

https://fluentbit.io

From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).

From this (https://github.com/fluent/fluent-bit/issues/723), I can see there is no grok support for fluent-bit.

apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: newrelic labels: k8s-app: newrelic-logging data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level ${LOG_LEVEL} Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE output-newrelic.conf @INCLUDE filter-kubernetes.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path ${PATH} Parser ${LOG_PARSER} DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name multiline Match * multiline.parser multiline-regex [FILTER] Name record_modifier Match * Record cluster_name ${CLUSTER_NAME} [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log Off output-newrelic.conf: | [OUTPUT] Name newrelic Match * licenseKey ${LICENSE_KEY} endpoint ${ENDPOINT} parsers.conf: | # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [MULTILINE_PARSER] name multiline-regex key_content message type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|--------------------------------|----------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"

There are also log forwarding tools like promtail and fluentbit that can be used to both ship logs to something like Loki and produce metrics.

FluentBit, is a logging processor that can help you to push all of your application logs to a central location like an ElasticSearch or OpenSearch cluster.

Been a while since I looked into fluentd and elasticsearch but I heard Stanza (soon to be moved to the OpenTelemetry Collector) is meant to be a modern replacement for it, but seems it's still in it's early days.

And then they contacted some famous log collection projects such as the Fluent Bit and also others like Stanza as well as Syslog, I guess. And I didn't read the whole thread of the conversation around log collection. But now, they set the Stanza as the first implementation of OpenTelemetry logs. And in Stanza, the observIQ with Stanza is merged under the OpenTelemetry log repository, so that's the status. And then, they try to standardize the format of logs based on the Stanza format, such as what kind of information should be included in log.

observIQ is a hosted log management platform that's powered by our FOSS log agent, Stanza: https://github.com/observIQ/stanza. Stanza is lightweight and robust (written in Go), and can be configured to act as a Syslog receiver in about a minute (guided configuration in our app).

observIQ is a simple, cost-effective hosted log management platform designed with small teams in-mind. Log collection is powered by Stanza: https://github.com/observIQ/stanza , our high-performance OSS log agent. Stanza was also recently adopted as the official log agent of the CNCF's OpenTelemetry project. observIQ offers guided-installation, pre-made dashboards, threshold-based alerts, live tail and more.

We're a new hosted log management platform with simple, native support for gathering logs from Openshift. Deployment typically takes a few minutes. We also automatically enrich Openshift logs metadata like node, namespace, pod, container - no additional configuration required. Logs are gathered with your high-performance OSS log agent, Stanza: https://github.com/observIQ/stanza

We offer a super simple, hosted log management platform that allows you to easily ingest any syslog traffic by setting up high-speed OSS agent as a syslog receiver in just a few click (setup takes about 30 seconds). You can find more about our log agent here: https://github.com/observIQ/stanza

Self-promotion alert, my apologies: Check us out at observIQ: https://observiq.com/ We provide a super-simple host log management platform powered our high-performance OSS Log Agent, Stanza (https://github.com/observIQ/stanza). Average setup time is about 5 minutes, end-to-end. Our platform is designed with start-ups and small businesses in-mind and are priced about half as much as our competitors. Lastly, we offer 14-day free trial, but also offer a completely free plan that provides 3 gigs of ingestion/day and 3 day retention - with access to the full feature set of the platform (Dashboards, Alerts, Live Tail). Shoot me a message or chat if you have any questions!

Similarly, our platform will be rolling out this feature soon, and will utilize our lightweight FOSS log agent Stanza - written in Go. The agent may work for your project https://github.com/observIQ/stanza