Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems. Learn more →
Top 23 Python Pipeline Projects
-
-
Judoscale
Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.
-
Project mention: Show HN: Flow – A Dynamic Task Engine for AI Agents Without DAG | news.ycombinator.com | 2024-12-02
- https://github.com/PrefectHQ/prefect
-
Project mention: Top 40 Open-source Developer Tools with the Most GitHub Stars | dev.to | 2025-04-20
GitHub: https://github.com/Avaiga/taipy
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
marimo
A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. All in a modern, AI-native editor.
linky https://github.com/marimo-team/marimo#:~:text=all%20in%20a%2... (Apache 2)
-
-
Kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Project mention: 20 Open Source Tools I Recommend to Build, Share, and Run AI Projects | dev.to | 2024-11-13Kedro is an ML development framework that brings data science projects from pilot development to production by creating reproducible, maintainable, and modular data science code. Kedro has a data catalog for data handling, support pipeline building, and a standardized template for code maintainability and consistency to effectively do this. Its data catalog uses lightweight data connectors to manage and track datasets. This allows you to use the same pipeline to build multiple production-level codes across your system.
-
CodeRabbit
CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
Here, we use the free Mage Ai orchestration tool.
-
-
AutoRAG
AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
📎 AutoRAG with Milvus 🛠️ ADO 🫶 Self Hosting LLM 🌐 Noema Declarative AI 📝 New NIM Blueprint for building AI Virtual Assistant 🚙 Zilliz Integrations 🫶 Using Milvus for Semantic Search 🤖 Contextual Retrieval 📎 Meta: Quantized Light Weight Models 🚙 https://arxiv.org/pdf/2407.01219 ✅ Cool Icons 🙌 IBM Watson AI Milvus Bot 📎 The Hacker's Browser 🛠️ Small and Mighty H2O Model 📝 Zilliz Cloud vs Qdrant 💫 Gravatino and Agents 🛠️ OSS Summit Europe 2024 Report ▶️ RAG Strategi 🤖 MS AI Data Visualizations 🌐 Graph RAG 👽 South Bay Meetup 15 Oct 2024 🦾 Influx and Milvus 👽 Multimodal Pipelines ✨ Constrained Sampling from LLM 🚕 BAML: Cheaper, Fast and More Accurate Function Calling 📊 Infinite World Generation with outlines txt 💻 Ollama Client Swift 🍔 Atomic Agents 🕶️ PYMUPDF4LLM 🚕 Milvus for AI Agents 📊 Fine Tuning LLAMA 3 with ORPO 🦾 Run NVIDIA Models 💻 LLM Training Meta Lingua ✨ 1 Bit LLM - MS BitNet 💻 Intro 🕶️ Mastering Chunk 📊 Storm Stanford Tool 🐍 DAMO NLP SG CaRing 🍔 LLM Reasoners
-
-
towhee
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
-
-
mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
-
-
-
-
-
toil
A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
-
NeumAI
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
-
koheesio
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
koheesio – framework for building efficient data pipelines
-
pypyr automation task runner
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
-
aws-lambda-handler-cookbook
This repository provides a working, deployable, open source-based, serverless service blueprint with an AWS Lambda function and AWS CDK Python code with all the best practices and a complete CI/CD pipeline.
The ‘orders’ service allows users to order products. We will use my open-source Serverless template project: AWS Lambda Handler Cookbook.
-
InfluxDB
InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Pipeline discussion
Python Pipeline related posts
-
Personal Picks: Data Product News (April 16, 2025)
-
airbyte VS cocoindex - a user suggested alternative
2 projects | 1 Apr 2025 -
Top 17 DevOps AI Tools [2025]
-
Can AI finally generate best practice code? I think so.
-
Jupyter Notebooks as E2E Tests
-
Explorer l'API de 360Learning : de l'agilité de Power Query à la robustesse de la Modern Data Stack
-
Airbyte 1.0 Released
-
A note from our sponsor - InfluxDB
influxdata.com | 25 Apr 2025
Index
What are some of the best open-source Pipeline projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | serve | 21,523 |
| 2 | Prefect | 19,016 |
| 3 | Taipy | 17,996 |
| 4 | airbyte | 17,903 |
| 5 | marimo | 12,524 |
| 6 | great_expectations | 10,334 |
| 7 | Kedro | 10,276 |
| 8 | Mage | 8,264 |
| 9 | papermill | 6,137 |
| 10 | AutoRAG | 3,856 |
| 11 | pipelines | 3,802 |
| 12 | towhee | 3,359 |
| 13 | PyFunctional | 2,420 |
| 14 | mara-pipelines | 2,082 |
| 15 | pytorch-toolbelt | 1,537 |
| 16 | MLBox | 1,503 |
| 17 | galaxy | 1,483 |
| 18 | sematic | 986 |
| 19 | toil | 909 |
| 20 | NeumAI | 854 |
| 21 | koheesio | 635 |
| 22 | pypyr automation task runner | 627 |
| 23 | aws-lambda-handler-cookbook | 610 |
Sponsored
Save 47% on cloud hosting with autoscaling that just works
Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.
judoscale.com