Open source contributions for a Data Engineer?

1. quinn

   1 9 670 6.2 Python

   pyspark methods to enhance developer productivity 📣 👯 🎉 (by mrpowers-io)

   

   I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. chispa

   2 12 691 7.7 Python

   PySpark test helper methods with beautiful error messages

   

   I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

4. spark-daria

   3 4 759 8.1 Scala

   Essential Spark extensions and helper methods ✨😲

   

   I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

5. spark-fast-tests

   4 6 447 8.5 Scala

   Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)

   

   I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.

6. soda-sql

   5 25 50 8.2 Python

   Discontinued Data profiling, testing, and monitoring for SQL accessible data.

   

   If you are interested in using/learning Python, SQL and data warehouse skills, take a look at https://github.com/sodadata/soda-sql

7. ballista

   6 20 2,238 9.3 Rust

   Discontinued Distributed compute platform implemented in Rust, and powered by Apache Arrow.

   

   His newer project, Ballista, was also donated to Apache Arrow. I hope to get the Rust skills to collaborate with him on open source work someday too. He's also doing really cool work on spark-rapids FYI.

8. spark-rapids

   7 6 898 9.8 Scala

   Spark RAPIDS plugin - accelerate Apache Spark with GPUs

   

   His newer project, Ballista, was also donated to Apache Arrow. I hope to get the Rust skills to collaborate with him on open source work someday too. He's also doing really cool work on spark-rapids FYI.

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

10. airbyte

    8 154 18,170 10.0 Python

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    

    Airbyte and Singer/Meltano if you want to learn more about ingestion pipelines. Airbyte and Meltano teams are very welcoming. SQLfluff a shiny SQL linter. Beautiful project with awesome maintainers.

11. meltano

    9 10 - -

    Airbyte and Singer/Meltano if you want to learn more about ingestion pipelines. Airbyte and Meltano teams are very welcoming. SQLfluff a shiny SQL linter. Beautiful project with awesome maintainers.

12. sqlfluff

    10 35 8,871 9.7 Python

    A modular SQL linter and auto-formatter with support for multiple dialects and templated code.

    

    Airbyte and Singer/Meltano if you want to learn more about ingestion pipelines. Airbyte and Meltano teams are very welcoming. SQLfluff a shiny SQL linter. Beautiful project with awesome maintainers.

13. DataGristle

    11 5 138 0.0 Python

    Tough and flexible tools for data analysis, transformation, validation and movement.

    

    DataGristle by u/kenfar who influenced many of us in this sub.

14. Metabase

    12 82 42,050 10.0 Clojure

    The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

    

    If you want to work more on the visualization side maybe Metabase, Superset and Streamlit.

15. superset

    13 149 66,270 9.9 Jupyter Notebook

    Apache Superset is a Data Visualization and Data Exploration Platform

    

    If you want to work more on the visualization side maybe Metabase, Superset and Streamlit.

16. streamlit

    14 310 39,392 9.9 Python

    Streamlit — A faster way to build and share data apps.

    

    If you want to work more on the visualization side maybe Metabase, Superset and Streamlit.

17. Skytrax-Data-Warehouse

    15 1 131 0.0 Python

    Discontinued A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

    Always open to accept contributions to my project (Skytrax Data Warehouse). If you are into data stuff support my work at youtube as well (One Developer Pirate), I mostly make data-oriented videos. These days I'm making a SQL course from a data analysis perspective that is expected to release in next week.

18. Prefect

    16 20 19,300 10.0 Python

    The easiest way to build, run, and monitor data pipelines at scale.

    

    Prefect! Specifically the Task Library: https://github.com/PrefectHQ/prefect

19. dagster

    17 52 13,154 10.0 Python

    An orchestration platform for the development, production, and observation of data assets.

    

    It's a near crime that Dagster hasn't been mentioned already.

20. SaaSHub

    www.saashub.com featured

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    

Suggest a related project