spark-operator VS charts

I have spent days now trying to figure out a dependency issue I'm experiencing with (Py)Spark running on Kubernetes. I'm using the spark-on-k8s-operator and Spark's Google Cloud connector.

We're using https://github.com/bitnami/charts/tree/main/bitnami/spark, but I have heard good things about https://github.com/GoogleCloudPlatform/spark-on-k8s-operator as well. Hudi should not need any long running deployments as per the docs https://hudi.apache.org/docs/0.5.1/deployment/#deploying

The SparkOperator must be installed before we can use Spark on Kubernetes. Google created this operator, which is available on Github. In a nutshell, the operator is in charge of monitoring the cluster for specific events related to the spark job, as known as kind: SparkApplication

Spark-Operator (as I understand, Spark-Operator integrates well with K8s while vanilla Spark integration with K8s seems extremely complex to maintain)

Spark on K8s Operator is a project from Google that allows submitting spark applications on Kubernetes cluster using CustomResource Definition SparkApplication. It uses mutating admission webhook to modify the pod spec and add the features not officially supported by spark-submit.

In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

Additional details of how SparkApplications are run can be found in the design documentation.

GoogleCloudPlatform/spark-on-k8s-operator (Go): Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Both of these examples seem pretty obvious and something you wouldn’t mess up, but as your chart grows, so does your values.yaml file. A great example is the Redis chart by Bitnami. I encourage you to scroll through its values file. See you in a minute!

We will write a Terraform module that will take a list of configurations for each required RabbitMQ instance. Luckily for us, we don't have to write the Kubernetes yaml configurations since the helm charts by Bitnami does a great job of doing all the things we discussed above. All we need to do is leverage Terraform Helm Provider and deploy the chart with the required values for our use case.

💡  The full details of helm charts can be referenced in their associated GitHub Repository.

I have a configmap with the ndjson set up under data:, similar to https://github.com/bitnami/charts/issues/6159 and it's subsequent answer.

This way, you can easily deploy any Helm charts from this public repo - https://github.com/bitnami/charts/tree/main/bitnami in just minutes.

The Bitnami Github Pages themselves usually cover everything you need to know. Configure a values.yaml file, or modify that to your liking, and you run helm install, as written in their docs.

The actual reason that our pods are not coming up is found when we review the helm installation that we are trying to run. If you check the dependencies in the GitHub repository (https://github.com/bitnami/charts/blob/main/bitnami/drupal/values.yaml) you find out that persistent storage is enabled by default and set to 8Gi. Also, the helm package uses MariaDB and the database size is specified to a default of 8Gi, thus setting the minimum storage for this installation to be 16Gi.

We're using https://github.com/bitnami/charts/tree/main/bitnami/spark, but I have heard good things about https://github.com/GoogleCloudPlatform/spark-on-k8s-operator as well. Hudi should not need any long running deployments as per the docs https://hudi.apache.org/docs/0.5.1/deployment/#deploying

Bitnami has its own scaffolding published at https://github.com/bitnami/charts/tree/main/template