Top 10 Python dataengineering Projects

dataengineering

• Add a project

1. sqlmesh

   1 12 2,256 9.9 Python

   Scalable and efficient data transformation framework - backwards compatible with dbt.

   

2. Judoscale

   judoscale.com featured

   Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

   

3. grai-core

   2 6 303 8.2 Python

   

4. beginner_de_project_stream

   3 1 100 3.2 Python

   Simple stream processing pipeline

   

5. pyspark-on-aws-emr

   4 1 26 3.6 Python

   The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.

   

6. data-engineer-challenge

   5 3 25 1.8 Python

   Challenge Data Engineer

   

7. pyDag

   6 2 24 0.0 Python

   Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

   

8. metadata-guardian

   7 1 16 5.1 Python

   Provide an easy way with Python to protect your data sources by searching its metadata. 🛡️

   

9. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

10. athenaSQL

    8 2 6 6.8 Python

    SQL builder for AWS Athena, inspired by sparkSQL

    

    Project mention: AthenaSQL: SQL query builder for AWS Athena, inspired by pySpark SQL | news.ycombinator.com | 2024-11-04

    Hi Everyone,

    I work in adtech, where we handle massive log-level data. To cut costs and improve performance for ML and optimization, my team and I chose a lakehouse approach using AWS (S3 + OTFs / partitioned Parquet + Athena + Glue).

    One challenge we faced with this data stack was managing Athena queries in our ETL jobs. Since Athena handles much of our data-heavy processing, we ended up storing hundreds of lines of query code as strings in Python scripts, which quickly became a nightmare to maintain.

    We needed something similar to PySpark SQL that could output SQL string compatible with Athena. So we built athenaSQL. It mimics the PySpark SQL API, providing a familiar interface and outputting SQL queries directly.

    It is far from complete at the moment but it has most of the basic query statements. I would love it if you could test it out and share any feedback! I hope someone is in need of such a tool, if it lacks the functionality you are seeking, let’s build it together! And feel free to critique it as much as you like. :)

    github: https://github.com/nabilseid/athenaSQL

    docs: github.com/nabilseid/athenaSQL

11. ticker_selection_BI_dashboard

    9 2 4 10.0 Python

    Data Engineering Project: 4 shares of a stock data extraction, upload on MySql used to be in a BI project

    

12. livyc

    10 2 3 0.0 Python

    Apache Spark as a Service with Apache Livy Client

    

Python dataengineering related posts

• The Data Engineering Handbook

  2 projects | news.ycombinator.com | 19 Nov 2024

Index

Sponsored

Save 47% on cloud hosting with autoscaling that just works

Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

judoscale.com

Do not miss the trending Python projects with our weekly report!