rq VS huey

That's pretty cool. Reckon it would work with existing code that calls Redis over the wire for RQ?

  https://python-rq.org

https://github.com/rq/rq is to the rescue.

We use RQ[0], it has Redis as a dependency. It’s pretty straightforward and we’re very happy with it. If you are using Django you may want to look at Django RQ[1] as well. RQ has built in scheduling capabilities these days, but historically it did not so we used (and still use) RQ Scheduler[2] which I think still has some advantages over the built in stuff.

[0] https://python-rq.org/

Also had a similar experience using RabbitMQ with Django+Celery. Extremely complicated and workers/queues would just stop for no reason.

Moved to Python-RQ [1] + Redis and been rock solid for years now.

[1] https://python-rq.org/

Yep, rq is simple and good: https://python-rq.org/ It also has a Django wrapper: https://github.com/rq/django-rq

We have a GPU (G5) instance that uses Python RQ (https://python-rq.org/).

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is created by docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

____

¹ https://github.com/coleifer/huey/

Other options are DjangoQ and Huey, which tend to work ok. Of the two I prefer DjangoQ. Database backed, don't require the Redis/Celery rigmarole.

Funny, just this moment i finally switched from Celery to huey. And so far I don't regret. huey looks very promising, has good documentation and is well integrated into DJango. You should give it a try: https://github.com/coleifer/huey

huey – a little task queue for python

Huey -> Same again?

Huey: https://github.com/coleifer/huey

I just want to add that a couple celery alternatives worth looking at include huey and dramatiq.

huey

same, I ran into an issue cos of django-background-tasks. I am thinking to replace it with huey