Spark open source community is awesome

• mack

  5 269 5.9 Python

  Delta Lake helper methods in PySpark

a couple devs just added a `find_compositite_keys_candidates` function so users can easily identify columns that could be used as a unique identifier in their Delta table.

• jodie

  2 44 4.9 Scala

  Delta lake and filesystem helper methods (by MrPowers)

another dev is working on adding an elegant interface to perform Hadoop filesystem operations, similar to os-lib for regular filesystem operations

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• os-lib

  4 640 7.5 Scala

  OS-Lib is a simple, flexible, high-performance Scala interface to common OS filesystem and subprocess APIs

  

another dev is working on adding an elegant interface to perform Hadoop filesystem operations, similar to os-lib for regular filesystem operations

• chispa

  12 508 6.7 Python

  PySpark test helper methods with beautiful error messages

here's a little README fix a user pushed to chispa

• delta-rs

  28 1,820 9.7 Rust

  A native Rust library for Delta Lake, with bindings into Python

  

Yea, there are tons of employees from companies that have made massive contributions to the Spark ecosystem. Apple built Delta Lake with Databricks, see this video for more detail. Lots of Spark PMCs are from various companies. delta-rs was initially built by Scribd and is now actively maintained by engineers at Voltron & other companies. It's awesome the community has so many contributors from various sources.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project