stable-fluids VS celery

For the lazy like me: https://github.com/GregTJ/stable-fluids

Created using a work-in-progress version of my Python based fluid simulator (unpublished, though the current version can be found here: https://github.com/GregTJ/stable-fluids)

One thing to note is if you need a response from the external API or not. If you don't, you can use a task queue (ex. Celery) which allows you to control task execution (ie. external API calls) rates and implement backoff strategies effortlessly. This allows you to eventually make all the calls you want, so long as the timing they get made isn't important. If you do require a response from the API - you can switch to a Websocket approach, where responses will be streamed back to the client (from the gateway) once the calls are made.

I came across this GitHub issue discussing PostgreSQL as a potential Celery broker. This led me to discover an interesting alternative task queue called BQ, which is built specifically to use PostgreSQL for task management.

The original implementation ran a Celery task that deleted notifications older than 90 days on a daily basis (original feature PR).

There are some limitations when using Quorum Queues, this was only patched recently - https://github.com/celery/celery/issues/9149

Celery previously had an --autoreload flag that has since been removed. However, Django has automatic reloading built into its manage.py runserver command. The absence of automatic reloading in Celery workers creates a confusing development experience: updating Python code causes the Django server to reload with the current code, but any tasks that the server fires will run stale code in the Celery worker.

From what I've seen, there are sort of two paths. I'll provide a well known example from each.

1. lang specific distributed task library

For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

* https://github.com/celery/celery

Or lower level:

* https://github.com/dask/dask

2. DAG Workflow systems

There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

* https://github.com/apache/airflow

https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365

https://docs.celeryq.dev/en/stable/getting-started/backends-...

Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol.

RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia

SQLite is written in C and has lots of tests because aerospace IIUC.

There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http

celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport...

Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-12...