SaaSHub helps you find the best software and product alternatives Learn more →
Top 20 Python Orchestration Projects
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
ragflow
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
-
-
covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)
-
-
-
prefect-deployment-patterns
Code examples showing flow deployment to various types of infrastructure
-
prism
Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python. (by runprism)
-
magniv-core
Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.
-
-
-
blueprint-examples
This is where you can find officially supported Cloudify blueprints that work with the latest versions of Cloudify. Please make sure to use the blueprints from this repo when you are evaluating Cloudify.
-
py-inventa
A Python library for microservice registry and executing RPC (Remote Procedure Call) over Redis.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions | dev.to | 2024-02-12Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.
Project mention: Prefect: A workflow orchestration tool for data pipelines | news.ycombinator.com | 2024-03-13
I’ve been working in tech for more than five years. I started as a Data Scientist, and now I’m exploring and loving the DevRel 🥑 role for Taipy. Needless to say, evolving in the tech scene has been a ride full of ups, downs, and everything in between.
Project mention: A mage on the Hero’s Journey: a fantasy epic on how a startup rose from the ashes | dev.to | 2023-06-12In the coming years, Mage will create a cooperative experience so that developers can build data pipelines with their team and level up together. After that journey, Mage will go on an epic quest to create the 1st open world community experience in the data universe.
Project mention: RAGFlow is an open-source RAG engine based on deep document understanding | news.ycombinator.com | 2024-04-01Just link them to https://github.com/infiniflow/ragflow/blob/main/rag/llm/chat... :)
Project mention: Kubernetes Enthusiasts: Share Your Ideas for Future Dev Tools | news.ycombinator.com | 2023-05-24https://github.com/kapicorp/kapitan is also a very powerful option for managing and generating templates.
Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like
Yeh - we have a small language list here:
https://github.com/nanovms/ops-examples
If you're looking for a particular piece of software search on the repo first:
If you don't find it and need help creating one ping me or open an issue in ops.
Project mention: Prism: the easiest way to create robust data workflows. Accessible via CLI | /r/coolgithubprojects | 2023-09-21
Project mention: Oakestra: A Lightweight Hierarchical Orchestration Framework for Edge Computing | news.ycombinator.com | 2024-03-03
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2024-04-01.
Python Orchestration related posts
- I may be the only evil (bit) user on the internet
- Is it ok to merge few applications into one ? How to do it ?
- How can one send an SSM command to run commands/scripts programmatically with Python CDK?
- Deploy a Django App on AWS Lightsail: Docker, Docker Compose, PostgreSQL, Nginx & Github Actions
- Kapitan: Generic templated configuration management for Kubernetes, Terraform
- Palo Alto Ansible benefits
-
A note from our sponsor - SaaSHub
www.saashub.com | 19 Apr 2024
Index
What are some of the best open-source Orchestration projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Airflow | 34,397 |
| 2 | jina | 19,884 |
| 3 | Prefect | 14,512 |
| 4 | dagster | 10,114 |
| 5 | Taipy | 8,257 |
| 6 | Mage | 6,953 |
| 7 | faraday | 4,600 |
| 8 | ragflow | 4,569 |
| 9 | kapitan | 1,742 |
| 10 | covalent | 687 |
| 11 | flintrock | 630 |
| 12 | esdc-ce | 114 |
| 13 | ops-examples | 96 |
| 14 | prefect-deployment-patterns | 93 |
| 15 | prism | 79 |
| 16 | magniv-core | 73 |
| 17 | oakestra | 44 |
| 18 | paco | 32 |
| 19 | blueprint-examples | 27 |
| 20 | py-inventa | 7 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com