couler
community
| couler | community | |
|---|---|---|
| 1 | 1 | |
| 890 | 152 | |
| 1.2% | 1.3% | |
| 5.2 | 7.7 | |
| 20 days ago | 2 days ago | |
| Python | Jsonnet | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
couler
-
(Not) to Write a Pipeline
author seems to be describing the kind of patterns you might make with https://argoproj.github.io/argo-workflows/ . or see for example https://github.com/couler-proj/couler , which is an sdk for describing tasks that may be submitted to different workflow engines on the backend.
it's a little confusing to me that the author seems to object to "pipelines" and then equate them with messaging-queues. for me at least, "pipeline" vs "workflow-engine" vs "scheduler" are all basically synonyms in this context. those things may or may not be implemented with a message-queue for persistence, but the persistence layer itself is usually below the level of abstraction that $current_problem is really concerned with. like the author says, eventually you have to track state/timestamps/logs, but you get that from the beginning if you start with a workflow engine.
i agree with author that message-queues should not be a knee-jerk response to most problems because the LoE for edge-cases/observability/monitoring is huge. (maybe reach for a queue only if you may actually overwhelm whatever the "scheduler" can handle.) but don't build the scheduler from scratch either.. use argowf, kubeflow, or a more opinionated framework like airflow, mlflow, databricks, aws lamda or step-functions. all/any of these should have config or api that's robust enough to express rate-limit/retry stuff. almost any of these choices has better observability out-of-the-box than you can easily get from a queue. but most importantly.. they provide idioms for handling failure that data-science folks and junior devs can work with. the right way to structure code is just much more clear and things like structuring messages/events, subclassing workers, repeating/retrying tasks, is just harder to mess up.
community
-
How can we read variables from file and use them in promql?
However I am not able to figure it out, how can I feed the string xyz_stack_1 to grafana. I have setup docker-compose.yaml file to start up all the containers. The configuration is done through prometheus.yaml, grafana.ini, dashboards.yaml and datasources.yaml
What are some alternatives?
soopervisor - ☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
kserve - Standardized Serverless ML Inference Platform on Kubernetes
argo - Workflow Engine for Kubernetes
pipelines - Machine Learning Pipelines for Kubeflow
awesome-argo - A curated list of awesome projects and resources related to Argo (a CNCF graduated project)
elyra - Elyra extends JupyterLab with an AI centric approach.
hera - Hera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo. ⭐️ Remember to star!
kfserving - Standardized Serverless ML Inference Platform on Kubernetes [Moved to: https://github.com/kserve/kserve]
prometheus - A docker-compose stack for Prometheus monitoring
argo-workflows-aws-plugin - Argo Workflows Executor Plugin for AWS Services, e.g. SageMaker Pipelines, Glue, etc.
kubeflow - Machine Learning Toolkit for Kubernetes