falcon VS Airflow

Falcon FastAPI Sanic Starlite (disclosure: I do work here)

Sanic is very very popular with 16.6k stars, 1.5k forks, opencollective sponsors and a very active github. Falcon is more popular than japronto with 8.9k stars, 898 forks, opencollective sponsors and a very active github too. Despite Japronto been keeped as first place by TechEmPower, Falcon is a way better solution in general with performance similar to fastify an very fast node.js framework that hits 575k requests per second in this benchmark.

I prefer Falcon for kicking up an API.

Falcon

And here's a project that's mostly Python, and optionally uses Cython https://github.com/falconry/falcon

To help with the above two cases I would consider using a microframework, and the Python community provides many solutions. In my professional career I’ve had the opportunity to work with three very good alternatives to Django: Flask, Falcon, and Fast API. Flask is designed to be easy to use and extend. It follows the principles of minimalism and gives more control over the app. Choosing it, developers can use multiple types of databases, which is not easy to do in Django. We can also plug in our favorite ORM and use it without any risk of unpredictable app behavior. In contrast to Django, it’s easy to integrate NoSQL databases with Flask.

This may not be exactly what you asked for but I found contributing to open source projects really exposed me to different approaches I never would have considered and may not have fully grasped had I not had to actually dive into the code to solve an issue. Falcon is a great place to start and the guys are super friendly there.

https://falcon.readthedocs.io/en/stable/ https://fastapi.tiangolo.com/

Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

In week Two, I contributed to the Apache Airflow repository.

Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform features a web-based user interface and a command-line interface for managing and triggering workflows.

Airflow is the most widely used and well-known tool for orchestrating data workflows. It allows for efficient pipeline construction, scheduling, and monitoring.

AIRFLOW This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic!

I agree there are many options in this space. Two others to consider:

- https://airflow.apache.org/

- https://github.com/spotify/luigi

There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory…

You can try and contribute some new connectors/operators for workflow managers like Airflow or Airbyte