Top 20 Python Pipeline Projects

Pipelines

• Add a project

1. Taipy

   1 24 18,047 9.8 Python

   Turns Data and AI algorithms into production-ready web applications in no time.

   

   Project mention: Top 40 Open-source Developer Tools with the Most GitHub Stars | dev.to | 2025-04-20

   GitHub: https://github.com/Avaiga/taipy

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. Mage

   2 79 8,301 9.5 Python

   🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

   

   Project mention: Wk 3 Orchestration: MLOPs with DataTalks | dev.to | 2025-02-22

   Here, we use the free Mage Ai orchestration tool.

4. zenml

   3 35 4,571 9.8 Python

   ZenML 🙏: The bridge between ML and Ops. https://zenml.io.

   

   Project mention: Accelerating ML Development with DevPods and ModelKits | dev.to | 2025-01-28

   Seamless integration: Works with OCI-compliant registries (e.g., Docker Hub and Jozu Hub) and integrates with popular tools like HuggingFace, ZenML, and Git.

5. polyaxon

   4 9 3,634 7.9 Python

   MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

   

6. ploomber

   5 121 3,569 6.4 Python

   The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

   

7. meltano

   6 9 2,050 9.8 Python

   Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

   

8. elyra

   7 2 1,918 8.6 Python

   Elyra extends JupyterLab with an AI centric approach.

   

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

10. covalent

    8 4 826 7.1 Python

    Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)

    

11. azure-devops-cli-extension

    9 4 650 7.0 Python

    Azure DevOps Extension for Azure CLI

    

12. pypyr automation task runner

    10 6 627 5.9 Python

    pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

    

13. pipefunc

    11 6 357 9.7 Python

    Lightweight fast function pipeline (DAG) creation in pure Python for scientific workflows 🕸️🧪

    

    Project mention: PipeFunc: Ultra Simple DAG Pipelines in Python with 15µs Overhead for Science | news.ycombinator.com | 2024-12-22

    • Any scenario involving interconnected functions where performance and ease of use are important

    I'd appreciate any feedback, especially regarding performance, usability, and potential applications in different scientific domains.

    Links:

    Documentation: https://pipefunc.readthedocs.io

14. sparktorch

    12 1 339 2.5 Python

    Train and run Pytorch models on Apache Spark.

    

15. patterns-devkit

    13 5 108 2.9 Python

    Data pipelines from re-usable components

    

16. xontrib-pipeliner

    14 1 59 7.6 Python

    Let your pipe lines flow thru the Python code in xonsh.

    

17. pipeline-runner

    15 1 55 7.7 Python

    Tool to run Bitbucket pipelines locally

    

18. SmartPipeline

    16 1 27 7.5 Python

    A framework for rapid development of robust data pipelines following a simple design pattern

    

19. panda_patrol

    17 2 21 8.0 Python

    

20. orinoco

    18 1 11 6.4 Python

    Functional composable pipelines allowing clean separation of the business logic and its implementation

    

21. pipelines

    19 3 4 2.7 Python

    Create Async Processing Pipelines Quick! (by theboxahaan)

    

22. m42pl-core

    20 1 4 0.0 Python

    A data manipulation language with a focus on flexibility and simplicity.

    

23. SaaSHub

    www.saashub.com featured

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    

Python Pipelines related posts

• PipeFunc: Ultra Simple DAG Pipelines in Python with 15µs Overhead for Science

  3 projects | news.ycombinator.com | 22 Dec 2024

• This Week In Python

  5 projects | dev.to | 11 Oct 2024

• Pipefunc: Minimalist DAG-Based Pipeline Management in Pure Python

  1 project | news.ycombinator.com | 11 Sep 2024

• PipelineC Example: FM Radio Demodulation (FPGA SDR)

  2 projects | news.ycombinator.com | 3 Mar 2024

• Generate non-CPU FPGA circuits from a C-like language

  1 project | news.ycombinator.com | 24 Nov 2023

• What makes C, Verilog, Java, Python, etc. so different?

  1 project | /r/ECE | 8 Jun 2023

• What are your private FPGA projects and why?

  2 projects | /r/FPGA | 31 May 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 12 May 2025

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

InfluxDB – Built for High-Performance Time Series Workloads

InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

www.influxdata.com

Do not miss the trending Python projects with our weekly report!