Ballista: New approach for 2021

• Apache Arrow

  75 13,480 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

• Apache Spark

  101 38,320 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

Yes, I think to the extent that the open-source Spark has support for columnar data exchange. I think some/much of the work has been done in the last 2 years (see https://github.com/apache/spark/pull/24795/files), but I don't now to what extent one could completely build out the execution part in Spark 3.0 or 3.1.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• spark-rapids

  3 720 9.8 Scala

  Spark RAPIDS plugin - accelerate Apache Spark with GPUs

  

So, in my day job at NVIDIA, I work on the RAPIDS Accelerator for Apache Spark, which is an open-source plugin that provides GPU-acceleration for ETL workloads, leveraging the RAPIDS cuDF GPU DataFrame library.

Suggest a related project