SaaSHub helps you find the best software and product alternatives Learn more →
Top 21 Python Pipeline Projects
-
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)
-
-
pypyr automation task runner
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
-
PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
-
Data Flow Facilitator for Machine Learning (dffml)
The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.
-
-
SmartPipeline
A framework for rapid development of robust data pipelines following a simple design pattern
-
orinoco
Functional composable pipelines allowing clean separation of the business logic and its implementation
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
I’ve been working in tech for more than five years. I started as a Data Scientist, and now I’m exploring and loving the DevRel 🥑 role for Taipy. Needless to say, evolving in the tech scene has been a ride full of ups, downs, and everything in between.
Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06- One-click sharing powered by Ploomber Cloud: https://ploomber.io
Documentation: https://jupysql.ploomber.io
Note that JupySQL is a fork of ipython-sql; which is no longer actively developed. Catherine, ipython-sql's creator, was kind enough to pass the project to us (check out ipython-sql's README).
We'd love to learn what you think and what features we can ship for JupySQL to be the best SQL client! Please let us know in the comments!
Project mention: meltano VS cloudquery - a user suggested alternative | libhunt.com/r/meltano | 2023-06-02
Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like
Project mention: PipelineC Example: FM Radio Demodulation (FPGA SDR) | news.ycombinator.com | 2024-03-03Related: PipelineC: A C-like hardware description language (HDL):
https://github.com/JulianKemmerer/PipelineC
Project mention: Show HN: Data monitoring and profiling with 1 function call | news.ycombinator.com | 2023-12-13
Project mention: Show HN: SmartPipeline, robust and light data pipelines in Python | news.ycombinator.com | 2023-05-03
Project mention: GitHub Actions can be tested locally with act, forget about pushing 37 commits trying to fix your CI/CD pipelines 😅 | /r/opensource | 2023-05-16Repo here
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Pipelines related posts
- PipelineC Example: FM Radio Demodulation (FPGA SDR)
- Generate non-CPU FPGA circuits from a C-like language
- What makes C, Verilog, Java, Python, etc. so different?
- What are your private FPGA projects and why?
- DBT lays off 15% of their staff
- SQL Mesh - Auto DAG generation!!
- SQL Mesh - Auto DAG generation!!
-
A note from our sponsor - SaaSHub
www.saashub.com | 24 Apr 2024
Index
What are some of the best open-source Pipeline projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Taipy | 8,371 |
| 2 | Mage | 7,001 |
| 3 | zenml | 3,657 |
| 4 | polyaxon | 3,479 |
| 5 | ploomber | 3,369 |
| 6 | elyra | 1,770 |
| 7 | meltano | 1,587 |
| 8 | covalent | 687 |
| 9 | azure-devops-cli-extension | 609 |
| 10 | pypyr automation task runner | 568 |
| 11 | PipelineC | 541 |
| 12 | sparktorch | 334 |
| 13 | Data Flow Facilitator for Machine Learning (dffml) | 240 |
| 14 | patterns-devkit | 106 |
| 15 | xontrib-pipeliner | 56 |
| 16 | panda_patrol | 21 |
| 17 | SmartPipeline | 21 |
| 18 | pipeline-runner | 19 |
| 19 | orinoco | 11 |
| 20 | m42pl-core | 4 |
| 21 | pipelines | 3 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com