Spark is lit once again

• incubator-livy

  2 692 0.0 Scala

  Mirror of Apache livy (Incubating)

  

  Apache Livy - an open-source REST API for interacting with Apache Spark from anywhere.

• incubator-livy

  1 11 10.0 Scala

  Mirror of Apache livy (Incubating) (by jahstreet)

  Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

• SonarLint

  www.sonarlint.org

  sponsored

  Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  

• lighter

  2 15 9.0 Java

  REST API for Apache Spark on K8S

  

  Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

• sparkmagic

  1 1,122 5.1 Python

  Jupyter magics and kernels for working with remote Spark clusters

  

  Things get a bit more complicated on interactive sessions. We've created Sparkmagic compatible REST API so that Sparkmagic kernel could communicate with Lighter the same way as it does with Apache Livy. When a user creates an interactive session Lighter server submits a custom PySpark application which contains an infinite loop which constantly checks for new commands to be executed. Each Sparkmagic command is saved on Java collection, retrieved by the PySpark application through Py4J Gateway and executed.

• Apache Spark

  39 32,903 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

  Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

• Apache Hadoop

  12 12,591 9.8 Java

  Apache Hadoop

  

  Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

Suggest a related project