Spark is lit once again

1. incubator-livy

   1 4 915 4.9 Scala

   Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

   

   Apache Livy - an open-source REST API for interacting with Apache Spark from anywhere.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. incubator-livy

   2 1 14 0.0 Scala

   Mirror of Apache livy (Incubating) (by jahstreet)

   

   Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

4. lighter

   3 2 98 9.5 Java

   REST API for Apache Spark on K8S or YARN

   

   Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

5. sparkmagic

   4 4 1,355 7.1 Python

   Jupyter magics and kernels for working with remote Spark clusters

   

   Things get a bit more complicated on interactive sessions. We've created Sparkmagic compatible REST API so that Sparkmagic kernel could communicate with Lighter the same way as it does with Apache Livy. When a user creates an interactive session Lighter server submits a custom PySpark application which contains an infinite loop which constantly checks for new commands to be executed. Each Sparkmagic command is saved on Java collection, retrieved by the PySpark application through Py4J Gateway and executed.

6. Apache Spark

   5 123 41,441 10.0 Scala

   Apache Spark - A unified analytics engine for large-scale data processing

   

   Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

7. Apache Hadoop

   6 41 15,167 9.9 Java

   Apache Hadoop

   

   Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

Suggest a related project