django-rq VS flower

For simple stuff, I tend to use https://github.com/rq/django-rq. Although scheduling tasks there does not work well in my experience. If I need something to run by schedule, than better just plain cron.

Performing tests on student images can be a time-consuming process, taking several minutes. Application has to download the image from the registry, create a container, wait for the application to load, and perform the test scenarios. To efficiently utilize the available hardware and handle increased server demand during peak times, a task queue was implemented using django-rq.

Yep, rq is simple and good: https://python-rq.org/ It also has a Django wrapper: https://github.com/rq/django-rq

I use RQ (Redis Queue) and it's very cool! https://github.com/rq/django-rq https://python-rq.org/patterns/django/

I love the simplicity of this idea because for lots of sites the database works just fine as a queue backend and it reduces the amount of infrastructure needed. I currently use https://github.com/dabapps/django-db-queue for devmarks.io which also uses the database to store tasks instead of a dedicated queue infrastructure. `Django Q` also has an option to use the database, but I haven't tested it at all: https://django-q.readthedocs.io/en/latest/configure.html#orm. And if you are already running `redis` for your site, https://github.com/rq/django-rq is another option.

The one benefit of this package is that it is async-first which will be beneficial as Django continually adds in more async capabilities. Nice work! I'm looking forward to trying this out and seeing how it works!

https://github.com/rq/django-rq if you already have redis available or https://github.com/dabapps/django-db-queue for something even simpler. I’ve used both and vastly prefer them over celery.

if you use redis there is https://github.com/rq/django-rq as well

Use flower to monitor workers and tasks.

Not sure what your criteria for good observability ist, but flower served me well enough for this in the last couple of years.

From experience i would not use threads for this or any background jobs. I would use Celery or Flask-RQ2 to be your workers, you will also probably end up using them to run other tasks as you encounter the need for other jobs. They both use Redis as a broker and job store and you can use Redis for other things like caching and so many other useful features. I kind of like RQ2 more then Celery because its a little simpler but Celery has a lot more to offer, more features. RQ2 has rq-dashboard for monitoring jobs and Celery has Flower.