cronitor-airflow
stepwise
| cronitor-airflow | stepwise | |
|---|---|---|
| 1 | 2 | |
| 2 | 37 | |
| - | - | |
| 10.0 | 3.8 | |
| about 2 years ago | 10 months ago | |
| Python | Clojure | |
| MIT License | Eclipse Public License 1.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cronitor-airflow
-
Lessons Learned from Running Apache Airflow at Scale
Is anybody out there doing anything interesting with Airflow monitoring?
At my startup Cronitor we have an Airflow sdk* that makes it pretty easy to provision monitoring for each DAG, but essentially we are only monitoring that a DAG started on time and the total time taken.
* https://github.com/cronitorio/cronitor-airflow
stepwise
-
Simplifying Step Functions and Stepwise: Lessons Learned and a New Approach
Luckily, [Stepwise v1](https://github.com/Motiva-AI/stepwise) can handle all the Step Functions deployment tidbits already. v2 is mostly an interface makeover.
-
Lessons Learned from Running Apache Airflow at Scale
I feel you. That's why we wrote a little library on top of SFN so that we can program SFN with Clojure instead of YAML https://github.com/Motiva-AI/stepwise. Application code sits with SFN definition and SFN Tasks are automatically integrated as polling Activities from Clojure code.
Thoughtworks made a case for this distinction in https://martinfowler.com/articles/cant-buy-integration.html#...
What are some alternatives?
windmill - Open-source developer platform to turn scripts into workflows and UIs. Fastest workflow engine (5x vs Airflow). Open-source alternative to Airplane and Retool.
dbt-core - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
direktiv - Serverless Container Orchestration
toil - A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
orchest - Build data pipelines, the easy way 🛠️
magniv-core - Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.
proposals - Temporal proposals
ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️