fluent-bit
helmfile
Our great sponsors
fluent-bit | helmfile | |
---|---|---|
35 | 39 | |
5,310 | 4,022 | |
2.2% | - | |
9.8 | 0.0 | |
6 days ago | 12 months ago | |
C | Go | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fluent-bit
-
Observability at KubeCon + CloudNativeCon Europe 2024 in Paris
Fluentbit
-
Fluent Bit with ECS: Configuration Tips and Tricks
$ docker run --rm fluent-bit-dummy WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/12/24 16:06:59] [ info] [fluent bit] version=1.9.10, commit=557c8336e7, pid=1 [2023/12/24 16:06:59] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2023/12/24 16:06:59] [ info] [cmetrics] version=0.3.7 [2023/12/24 16:06:59] [ info] [output:stdout:stdout.0] worker #0 started [2023/12/24 16:06:59] [ info] [sp] stream processor started [0] dummy.0: [1703434019.553880465, {"message"=>"custom dummy"}] [0] dummy.0: [1703434020.555768799, {"message"=>"custom dummy"}] [0] dummy.0: [1703434021.550525174, {"message"=>"custom dummy"}] [0] dummy.0: [1703434022.551563050, {"message"=>"custom dummy"}] [0] dummy.0: [1703434023.551944509, {"message"=>"custom dummy"}] [0] dummy.0: [1703434024.550027843, {"message"=>"custom dummy"}] [0] dummy.0: [1703434025.550901801, {"message"=>"custom dummy"}] [0] dummy.0: [1703434026.549279385, {"message"=>"custom dummy"}] ^C[2023/12/24 16:07:08] [engine] caught signal (SIGINT) [0] dummy.0: [1703434027.549678344, {"message"=>"custom dummy"}] [2023/12/24 16:07:08] [ warn] [engine] service will shutdown in max 5 seconds [2023/12/24 16:07:08] [ info] [engine] service has stopped (0 pending tasks) [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopping... [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopped
-
Should You Be Scared of Unix Signals?
> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers.
And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory.
I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling.
-
Vector: a Rust-based lightweight alternative to Fluentd/Logstash
Fluentbit is Fluentd's lightweight alternative to itself.
- FLaNK Stack Weekly for 14 Aug 2023
-
Ultimate EKS Baseline Cluster: Part 1 - Provision EKS
From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).
-
Fluentbit Kubernetes - How to extract fields from existing logs
From this (https://github.com/fluent/fluent-bit/issues/723), I can see there is no grok support for fluent-bit.
-
Parsing multiline logs using a custom Fluent Bit configuration
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: newrelic labels: k8s-app: newrelic-logging data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level ${LOG_LEVEL} Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE output-newrelic.conf @INCLUDE filter-kubernetes.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path ${PATH} Parser ${LOG_PARSER} DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name multiline Match * multiline.parser multiline-regex [FILTER] Name record_modifier Match * Record cluster_name ${CLUSTER_NAME} [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log Off output-newrelic.conf: | [OUTPUT] Name newrelic Match * licenseKey ${LICENSE_KEY} endpoint ${ENDPOINT} parsers.conf: | # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [MULTILINE_PARSER] name multiline-regex key_content message type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|--------------------------------|----------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"
-
Tool to scrape (semi)-structured log files (e.g. log4j)
There are also log forwarding tools like promtail and fluentbit that can be used to both ship logs to something like Loki and produce metrics.
-
How to Deploy and Scale Strapi on a Kubernetes Cluster 2/2
FluentBit, is a logging processor that can help you to push all of your application logs to a central location like an ElasticSearch or OpenSearch cluster.
helmfile
-
Deploy IRIS Application to Azure Using CircleCI
What we’re going to install into the newly created AKS cluster is located in the helm directory. The descriptive Helmfile approach enables us to define applications and their settings in the helmfile.yaml file.
-
[2022] [Updated] Alternative to Helmfile
Is there any alternative to https://github.com/roboll/helmfile you are currently using in your company.
-
Projectsveltos: Manage Kubernetes addons in multiple clusters
Interesting, I have approached this problem using Helmfile (https://github.com/roboll/helmfile) to define a “platform release package.”
-
How are you handling ILM on kubernetes?
To make managing the Helm deployments a little easier I used helmfile (https://github.com/roboll/helmfile).
-
Helm Charts Microservices
But in general it's always easier to keep things quite separated. Meaning in separate helm releases. If you want to be able to manage things "together" at will, then you can use helmfile ( https://github.com/roboll/helmfile )
-
How to Build Software Like an SRE
I agree; helm is too declarative.
Whenever I can, I use helmfile[0] for storing variables for helm since it does add a declarative layer on top of helm.
-
helmfile sync vs helmfile apply
I went through the Helmfile repo Readme to figure out the difference between helmfile sync and helmfile apply. It seems like unlike the apply command, the sync command doesn't do a diff and helm upgrades the hell out of all releases 😃. But from the word sync, you'd expect the command to apply those releases that have been changed. There is also mention of the potential application of helmfile apply to periodically syncing of releases. Why not use helmfile sync for this purpose? Overall, the difference didn't become crystal clear, and I though there could probably be more to it. So, I'm asking.
-
Managing multiple repos
helmfile is something i’ve used in the past for this https://github.com/roboll/helmfile
-
Helm is both "package manager" and "templating engine" - probably the best package manager but horrible template engine
I always felt like dependencies in helm are for very simple non-coupled packages. I many times use Helmfile (https://github.com/roboll/helmfile) to manage dependencies instead of banging my head with vanilla Helm.
-
So I've installed grafana, loki, and prometheus on the personal Kubernetes cluster via Terraform. Now what?
Once you do that, learn to create dynamic helm charts that use go templating and conditionals: https://github.com/roboll/helmfile
What are some alternatives?
loki - Like Prometheus, but for logs.
flux2 - Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.
rsyslog - a Rocket-fast SYStem for LOG processing
cdk8s - Define Kubernetes native apps and abstractions using object-oriented programming
syslog-ng - syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
helmsman - Helm Charts as Code
jaeger - CNCF Jaeger, a Distributed Tracing Platform
kustomize - Customization of kubernetes YAML configurations
winston - A logger for just about everything.
helm-operator - Successor: https://github.com/fluxcd/helm-controller — The Flux Helm Operator, once upon a time a solution for declarative Helming.
Grafana - The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.