spark-operator VS fission

I have spent days now trying to figure out a dependency issue I'm experiencing with (Py)Spark running on Kubernetes. I'm using the spark-on-k8s-operator and Spark's Google Cloud connector.

We're using https://github.com/bitnami/charts/tree/main/bitnami/spark, but I have heard good things about https://github.com/GoogleCloudPlatform/spark-on-k8s-operator as well. Hudi should not need any long running deployments as per the docs https://hudi.apache.org/docs/0.5.1/deployment/#deploying

The SparkOperator must be installed before we can use Spark on Kubernetes. Google created this operator, which is available on Github. In a nutshell, the operator is in charge of monitoring the cluster for specific events related to the spark job, as known as kind: SparkApplication

Spark-Operator (as I understand, Spark-Operator integrates well with K8s while vanilla Spark integration with K8s seems extremely complex to maintain)

Spark on K8s Operator is a project from Google that allows submitting spark applications on Kubernetes cluster using CustomResource Definition SparkApplication. It uses mutating admission webhook to modify the pod spec and add the features not officially supported by spark-submit.

In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

Additional details of how SparkApplications are run can be found in the design documentation.

GoogleCloudPlatform/spark-on-k8s-operator (Go): Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

Fission

This is where I see K8S coming in – teachers can provide dev deployments that are setup for students to learn. Teachers can also provide containers that run automated tests against the student containers for assessment! Plus, we can smooth over some of the git workflow stuff for the ripest of beginners; we can integrate with github to sync their work on our platform to repositories on their github account, so that they can really take ownership of the work they do on the platform. Last, students can graduate their work from development into production very easily, since we can take the base images + student diffs, build a new "prod" image for the student. We can run students' prod work on "serverless" K8S frameworks like fission or OpenFaas to be able to host many low-traffic "production" apps at the same time.

I use https://fission.io/ on Kubernetes to emulate AWS Lambda + API Gateway to run Python functions. I use their YAML Spec functionality to deploy functions. It works well for my use case.

After doing a lot of research, I ended up settling on the Fission.io framework to support this project. Fission is an open-source Serverless framework running in kubernetes. Think AWS Lambdas, but we are in control of every part of the infrastructure. Kubernetes gives us the power to define the environments the containers will be executed in, and any other resources they need. This gives us the control we need to be able to create our very own environment for executing arbitrary JavaScript through the V8 engine. Each function can be isolated as much as we need to and Fission is really great at giving us the ability to quickly create multiple environments.

Nope. I was using https://fission.io/

There are a lot of FaaS service offerings out there namely Lambda, Azure Functions, Google Cloud Functions to name a few. While these offerings run on their respective clouds, there are services like Fission that are open source and allow you to deploy and execute functions on Kubernetes clusters irrespective of where they reside.

Fission

It's super slow (20-50 rps), please try this instead : https://github.com/fission/fission (few hundreds to few thousands rps)