5-minute-production-app VS fluent-bit

Author here.

This post doesn't live up to its title, I'm sorry about that.

The title should have been 'We are building a better Heroku FOR PRODUCTION APPS' (we'll add the 'for production apps' to the title)

It should have emphasized the _building_ part, we're just starting. The current 5 minute production app doesn't hold a candle to Heroku at the moment.

It should have made it clear the goals is to improve the speed with which you can configure a production app, not a development app. Development apps on Heroku are already close to perfect. The examples in this post are contrived since it talks about a development app, as rightly called out by Heroku people https://twitter.com/johnbeynon/status/1374306499426652161

It should have gone into why hyper clouds might be preferable https://gitlab.com/gitlab-org/5-minute-production-app/deploy...

It should have talked about state, we made a small improvement in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request... but we should have done the planned work in https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11137 and made one post out of it.

Disclaimer, I am a CNCF Ambassador (voluntary) - so it's in my interest to promote CNCF projects like Kubernetes.

It seems like a good time to mention my blog post from last year "Then he asked me “Is Kubernetes right for us?" -> https://alexellisuk.medium.com/then-he-asked-me-is-kubernete...

Some of the feedback I've had so far is that it was refreshing to get "permission" to consider alternatives vs. the current hype. I use K8s and K3s quite broadly myself, but increasingly see consulting prospects and customers who are not comfortable to make the leap, but are very happy on managed services with their chosen vendor - Azure / AWS / GCP.

GitLab recently released a bunch of terraform to show you how to run side projects in the free tier of a cloud - https://gitlab.com/gitlab-org/5-minute-production-app/deploy...

The OpenFaaS project again is very coupled to Kubernetes, making it easier to use and more reliable is important to the community and project's future. However, we created a version called faasd that works more like docker-compose. It's received much more traction than we expected and companies and individuals are putting it into production. It does't have clustering, and supports only 1 replica per function, so it's surprising.

I'll keep doing my bit to promote solutions that make K8s easier to understand like K3s (see also k3sup.dev) and to look into alternatives. But as you will see in my blog post - I don't think it's right to assume Kubernetes is the right solution for every team, and every project, without first talking about the problem being solved.

Less than five minutes. I worked closely with GitLab co-founders over the past 7 weeks, and we built the [Five Minute Production](https://gitlab.com/gitlab-org/5-minute-production-app/deploy-template/-/blob/master/README.md) project that generates AWS infra for webapps with minimal fuss.

Fluentbit

$ docker run --rm fluent-bit-dummy WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/12/24 16:06:59] [ info] [fluent bit] version=1.9.10, commit=557c8336e7, pid=1 [2023/12/24 16:06:59] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2023/12/24 16:06:59] [ info] [cmetrics] version=0.3.7 [2023/12/24 16:06:59] [ info] [output:stdout:stdout.0] worker #0 started [2023/12/24 16:06:59] [ info] [sp] stream processor started [0] dummy.0: [1703434019.553880465, {"message"=>"custom dummy"}] [0] dummy.0: [1703434020.555768799, {"message"=>"custom dummy"}] [0] dummy.0: [1703434021.550525174, {"message"=>"custom dummy"}] [0] dummy.0: [1703434022.551563050, {"message"=>"custom dummy"}] [0] dummy.0: [1703434023.551944509, {"message"=>"custom dummy"}] [0] dummy.0: [1703434024.550027843, {"message"=>"custom dummy"}] [0] dummy.0: [1703434025.550901801, {"message"=>"custom dummy"}] [0] dummy.0: [1703434026.549279385, {"message"=>"custom dummy"}] ^C[2023/12/24 16:07:08] [engine] caught signal (SIGINT) [0] dummy.0: [1703434027.549678344, {"message"=>"custom dummy"}] [2023/12/24 16:07:08] [ warn] [engine] service will shutdown in max 5 seconds [2023/12/24 16:07:08] [ info] [engine] service has stopped (0 pending tasks) [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopping... [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopped

> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers.

And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory.

I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling.

[1] https://github.com/fluent/fluent-bit/issues/4836

Fluentbit is Fluentd's lightweight alternative to itself.

https://fluentbit.io

From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).

From this (https://github.com/fluent/fluent-bit/issues/723), I can see there is no grok support for fluent-bit.

apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: newrelic labels: k8s-app: newrelic-logging data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level ${LOG_LEVEL} Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE output-newrelic.conf @INCLUDE filter-kubernetes.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path ${PATH} Parser ${LOG_PARSER} DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name multiline Match * multiline.parser multiline-regex [FILTER] Name record_modifier Match * Record cluster_name ${CLUSTER_NAME} [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log Off output-newrelic.conf: | [OUTPUT] Name newrelic Match * licenseKey ${LICENSE_KEY} endpoint ${ENDPOINT} parsers.conf: | # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [MULTILINE_PARSER] name multiline-regex key_content message type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|--------------------------------|----------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"

There are also log forwarding tools like promtail and fluentbit that can be used to both ship logs to something like Loki and produce metrics.

FluentBit, is a logging processor that can help you to push all of your application logs to a central location like an ElasticSearch or OpenSearch cluster.