toil
cronitor-airflow
toil | cronitor-airflow | |
---|---|---|
2 | 1 | |
870 | 2 | |
0.3% | - | |
9.2 | 10.0 | |
3 days ago | almost 2 years ago | |
Python | Python | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
toil
-
Show HN: Hatchet – Open-source distributed task queue
a little late now, but I wonder if https://github.com/DataBiosphere/toil might meet your requirements
- Lessons Learned from Running Apache Airflow at Scale
cronitor-airflow
-
Lessons Learned from Running Apache Airflow at Scale
Is anybody out there doing anything interesting with Airflow monitoring?
At my startup Cronitor we have an Airflow sdk* that makes it pretty easy to provision monitoring for each DAG, but essentially we are only monitoring that a DAG started on time and the total time taken.
* https://github.com/cronitorio/cronitor-airflow
What are some alternatives?
common-workflow-language - Repository for the CWL standards. Use https://cwl.discourse.group/ for support 😊
windmill - Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool.
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
stepwise - Clojure AWS Step Functions library
direktiv - Serverless Container Orchestration
states-language-cadence - States Language on Cadence
orchest - Build data pipelines, the easy way 🛠️
magniv-core - Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.