huey VS django-upgrade

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is created by docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

____

¹ https://github.com/coleifer/huey/

Other options are DjangoQ and Huey, which tend to work ok. Of the two I prefer DjangoQ. Database backed, don't require the Redis/Celery rigmarole.

Funny, just this moment i finally switched from Celery to huey. And so far I don't regret. huey looks very promising, has good documentation and is well integrated into DJango. You should give it a try: https://github.com/coleifer/huey

huey – a little task queue for python

Huey -> Same again?

Huey: https://github.com/coleifer/huey

I just want to add that a couple celery alternatives worth looking at include huey and dramatiq.

huey

same, I ran into an issue cos of django-background-tasks. I am thinking to replace it with huey

30% of Django developers upgrade every LTS release[1]. As Django 3.2 runs out of support in April this year[2], it's worth celebrating Adam Johnson's package django-upgrade (https://github.com/adamchainz/django-upgrade) which has a set of fixers to automatically update your Django code to the version of your choice.

Curious if anyone has used 2to3 (https://docs.python.org/3/library/2to3.html) for upgrading Python code and can share their experience with it? Would you recommend?

1. https://lp.jetbrains.com/django-developer-survey-2022/

Take a look at https://github.com/adamchainz/django-upgrade

I had easy and quick upgrade from 3.2 to 4.1 using django-upgrade package. Project was pretty big, and I didn’t expect to be that easy to migrate to 4.1.

Regarding the actual Django upgrade you might want to take a look at Django Upgrade https://github.com/adamchainz/django-upgrade - a tool to automate some of the boring tedious stuff in an upgrade.

Running https://github.com/adamchainz/django-upgrade with https://github.com/asottile/pyupgrade recursively will give an idea about how much work is there on Django side. Still, there may have dependency on third party libraries (both django+python). Another thing to consider is which role Django performing here, serving APIs or html views. As good test coverage is already there, you are on lucky side.

I recently upgraded one of my project to Django v4, from Django v1. The changes from Django 3 to 4 were fairly simple, just one line change in my case. The change log is extremely detailed, it was easy to upgrade. Also, I found a tool called django-upgrade which makes some changes automatically, rest I made manually.

Adam Johnson has written a nifty tool (https://github.com/adamchainz/django-upgrade) that can automatically do this for you. Even if you don't want to use the tool, the README generally outlines all the key issues you need to be aware of.

You should try the new [django-upgrade](https://github.com/adamchainz/django-upgrade) package. It claims you can either just run it or hook it into your CI. It will automatically take care of some of the heavy lifting for you.