My Journey With Spark On Kubernetes... In Python (1/3)

This page summarizes the projects mentioned and recommended in the original post on dev.to
#Kubernetes #Charts #Helm #batch-systems #Cncf

Our great sponsors
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SonarQube - Static code analysis for 29 languages.
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • volcano

    A Cloud Native Batch System (Project under CNCF)

    For our experiments, we will use Volcano which is a batch scheduler for Kubernetes, well-suited for scheduling Spark applications pods with a better efficiency than the default kube-scheduler. The main reason is that Volcano allows "group scheduling" or "gang scheduling": while the default scheduler of Kubernetes schedules containers one by one, Volcano ensures that a gang of related containers (here, the Spark driver and its executors) can be scheduled at the same time. If for any reason it is not possible to deploy all the containers in a gang, Volcano will not schedule that gang. This article explains in more detail the reasons for using Volcano.

  • helm

    The Kubernetes Package Manager [Moved to: https://github.com/helm/helm] (by kubernetes)

    In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.

  • spark-on-k8s-operator

    Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

    In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

  • charts

    ⚠️(OBSOLETE) Curated applications for Kubernetes (by helm)

    In this section, you use Helm to deploy the Kubernetes Operator for Apache Spark from the incubator Chart repository. Helm is a package manager you can use to configure and deploy Kubernetes apps.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts