Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 6 argo-workflow Open-Source Projects
-
awesome-argo
A curated list of awesome projects and resources related to Argo (a CNCF graduated project)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
couler
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
-
hera
Hera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo. ⭐️ Remember to star!
-
soopervisor
☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
-
argo-workflows-aws-plugin
Argo Workflows Executor Plugin for AWS Services, e.g. SageMaker Pipelines, Glue, etc.
Like Argo Workflows?
https://github.com/argoproj/argo-workflows
author seems to be describing the kind of patterns you might make with https://argoproj.github.io/argo-workflows/ . or see for example https://github.com/couler-proj/couler , which is an sdk for describing tasks that may be submitted to different workflow engines on the backend.
it's a little confusing to me that the author seems to object to "pipelines" and then equate them with messaging-queues. for me at least, "pipeline" vs "workflow-engine" vs "scheduler" are all basically synonyms in this context. those things may or may not be implemented with a message-queue for persistence, but the persistence layer itself is usually below the level of abstraction that $current_problem is really concerned with. like the author says, eventually you have to track state/timestamps/logs, but you get that from the beginning if you start with a workflow engine.
i agree with author that message-queues should not be a knee-jerk response to most problems because the LoE for edge-cases/observability/monitoring is huge. (maybe reach for a queue only if you may actually overwhelm whatever the "scheduler" can handle.) but don't build the scheduler from scratch either.. use argowf, kubeflow, or a more opinionated framework like airflow, mlflow, databricks, aws lamda or step-functions. all/any of these should have config or api that's robust enough to express rate-limit/retry stuff. almost any of these choices has better observability out-of-the-box than you can easily get from a queue. but most importantly.. they provide idioms for handling failure that data-science folks and junior devs can work with. the right way to structure code is just much more clear and things like structuring messages/events, subclassing workers, repeating/retrying tasks, is just harder to mess up.
Project mention: AWS services executor plugin for Argo Workflows | news.ycombinator.com | 2023-11-09
argo-workflows related posts
-
Ploomber Cloud - Parametrizing and running notebooks in the cloud in parallel
-
Show HN: Ploomber Cloud (YC W22) – run notebooks at scale without infrastructure
-
awesome-argo: A curated list of awesome projects and resources related to Argo (a CNCF hosted project)
-
terrytangyuan/awesome-argo: A curated list of awesome projects and resources related to Argo (a CNCF hosted project)
-
awesome-argo: A curated list of projects and resources related to Argo
-
awesome-argo: A curated list of awesome projects and resources related to Argo (a CNCF hosted project)
-
A note from our sponsor - InfluxDB
www.influxdata.com | 1 May 2024
Index
What are some of the best open-source argo-workflow projects? This list will help you:
Project | Stars | |
---|---|---|
1 | argo | 14,314 |
2 | awesome-argo | 1,792 |
3 | couler | 889 |
4 | hera | 484 |
5 | soopervisor | 42 |
6 | argo-workflows-aws-plugin | 4 |
Sponsored