Ask HN: What's the right tool for this job?

1. flyte

   1 39 6,192 9.7 Go

   Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

   

   My $0.02: https://flyte.org/ - you write the python functions, they take an s3 (or similar) path to the images, and flute handles the orchestration for you, also allowing you to control how much compute is thrown at the problem, which essentially gives you your queue.

   If cost of operations starts to be an issue you can start moving elements to your own infrastructure.

2. Judoscale

   judoscale.com featured

   Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

   

3. celery

   2 54 26,181 9.6 Python

   Distributed Task Queue (development branch)

   

   From what I've seen, there are sort of two paths. I'll provide a well known example from each.

   1. lang specific distributed task library

   For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

   * https://github.com/celery/celery

   Or lower level:

   * https://github.com/dask/dask

   2. DAG Workflow systems

   There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

   * https://github.com/apache/airflow

4. Dask

   3 33 13,148 9.7 Python

   Parallel computing with task scheduling

   

   From what I've seen, there are sort of two paths. I'll provide a well known example from each.

   1. lang specific distributed task library

   For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

   * https://github.com/celery/celery

   Or lower level:

   * https://github.com/dask/dask

   2. DAG Workflow systems

   There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

   * https://github.com/apache/airflow

5. Airflow

   4 187 39,794 10.0 Python

   Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

   

   From what I've seen, there are sort of two paths. I'll provide a well known example from each.

   1. lang specific distributed task library

   For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

   * https://github.com/celery/celery

   Or lower level:

   * https://github.com/dask/dask

   2. DAG Workflow systems

   There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

   * https://github.com/apache/airflow

Suggest a related project