Spark is lit once again

• incubator-livy

  4 892 5.4 Scala

  Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

  

  Apache Livy - an open-source REST API for interacting with Apache Spark from anywhere.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• incubator-livy

  1 15 0.0 Scala

  Mirror of Apache livy (Incubating) (by jahstreet)

  Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

• lighter

  2 92 9.4 Java

  REST API for Apache Spark on K8S or YARN

  

  Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

• sparkmagic

  4 1,331 8.1 Python

  Jupyter magics and kernels for working with remote Spark clusters

  

  Things get a bit more complicated on interactive sessions. We've created Sparkmagic compatible REST API so that Sparkmagic kernel could communicate with Lighter the same way as it does with Apache Livy. When a user creates an interactive session Lighter server submits a custom PySpark application which contains an infinite loop which constantly checks for new commands to be executed. Each Sparkmagic command is saved on Java collection, retrieved by the PySpark application through Py4J Gateway and executed.

• Apache Spark

  109 40,073 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

  Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

• Apache Hadoop

  29 14,801 9.8 Java

  Apache Hadoop

  

  Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

Suggest a related project