bitnami-docker-kafka VS charts

Regarding Kafka, try the Bitnami image, and see their GitHub page which has most of what you need to spin up a simple node: https://github.com/bitnami/bitnami-docker-kafka

View on GitHub

I am using Bitnami's Docker Image for Kafka. It is well documented and frequently updated. We will be doing a straight forward setup and add a firewall allowing only the private/public ips that belongs to our application servers.

Both of these examples seem pretty obvious and something you wouldn’t mess up, but as your chart grows, so does your values.yaml file. A great example is the Redis chart by Bitnami. I encourage you to scroll through its values file. See you in a minute!

We will write a Terraform module that will take a list of configurations for each required RabbitMQ instance. Luckily for us, we don't have to write the Kubernetes yaml configurations since the helm charts by Bitnami does a great job of doing all the things we discussed above. All we need to do is leverage Terraform Helm Provider and deploy the chart with the required values for our use case.

💡  The full details of helm charts can be referenced in their associated GitHub Repository.

I have a configmap with the ndjson set up under data:, similar to https://github.com/bitnami/charts/issues/6159 and it's subsequent answer.

This way, you can easily deploy any Helm charts from this public repo - https://github.com/bitnami/charts/tree/main/bitnami in just minutes.

The Bitnami Github Pages themselves usually cover everything you need to know. Configure a values.yaml file, or modify that to your liking, and you run helm install, as written in their docs.

The actual reason that our pods are not coming up is found when we review the helm installation that we are trying to run. If you check the dependencies in the GitHub repository (https://github.com/bitnami/charts/blob/main/bitnami/drupal/values.yaml) you find out that persistent storage is enabled by default and set to 8Gi. Also, the helm package uses MariaDB and the database size is specified to a default of 8Gi, thus setting the minimum storage for this installation to be 16Gi.

We're using https://github.com/bitnami/charts/tree/main/bitnami/spark, but I have heard good things about https://github.com/GoogleCloudPlatform/spark-on-k8s-operator as well. Hudi should not need any long running deployments as per the docs https://hudi.apache.org/docs/0.5.1/deployment/#deploying

Bitnami has its own scaffolding published at https://github.com/bitnami/charts/tree/main/template