Spark is lit once again

This page summarizes the projects mentioned and recommended in the original post on dev.to

Stream - Scalable APIs for Chat, Feeds, Moderation, & Video.
Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.
getstream.io
featured
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com
featured
  1. incubator-livy

    Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

    Apache Livy - an open-source REST API for interacting with Apache Spark from anywhere.

  2. Stream

    Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

    Stream logo
  3. incubator-livy

    Mirror of Apache livy (Incubating) (by jahstreet)

    Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

  4. lighter

    REST API for Apache Spark on K8S or YARN

    Latter was a go-to solution at the time when we were only using Spark on YARN. Sadly Apache Livy is not maintained anymore: it has no K8s support, Spark client is more and more outdated with every passing day. For some time we used @jahstreet's fork which had K8s available. But then we saw that the Livy project hadn't received any updates and we decided to implement our own solution - Exacaster Lighter.

  5. sparkmagic

    Jupyter magics and kernels for working with remote Spark clusters

    Things get a bit more complicated on interactive sessions. We've created Sparkmagic compatible REST API so that Sparkmagic kernel could communicate with Lighter the same way as it does with Apache Livy. When a user creates an interactive session Lighter server submits a custom PySpark application which contains an infinite loop which constantly checks for new commands to be executed. Each Sparkmagic command is saved on Java collection, retrieved by the PySpark application through Py4J Gateway and executed.

  6. Apache Spark

    Apache Spark - A unified analytics engine for large-scale data processing

    Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

  7. Apache Hadoop

    Apache Hadoop

    Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Unveiling the Apache License 2.0: A Deep Dive into Open Source Freedom

    3 projects | dev.to | 11 Mar 2025
  • How to Install PySpark on Your Local Machine

    2 projects | dev.to | 9 Dec 2024
  • Unveiling the Analytics Industry in Bangalore

    3 projects | /r/u_Khushisondhi7 | 23 Mar 2023
  • 5 Best Practices For Data Integration To Boost ROI And Efficiency

    3 projects | /r/ReviewNPrep | 12 Mar 2023
  • Big Data Processing, EMR with Spark and Hadoop | Python, PySpark

    2 projects | dev.to | 27 Mar 2022

Did you know that Scala is
the 32nd most popular programming language
based on number of references?