dramatiq VS django-q

Using something like Dramatiq [1] with Redis, writing a background job takes minutes, and can be deployed alongside an existing Python web app. There are probably JS equivalents.

I think Inngest could be a useful service, but the comparison felt off for me - it made me feel like this wasn't solving a real problem.

[1] https://dramatiq.io/

Dramatiq documentation

Hello everyone. We want to present you Taskiq: our new project that allows sending tasks using distributed queues. Conceptually it's similar to Celery or Dramatiq but with full asyncio and type hints support. Taskiq can send and execute async functions and has many integrations with different queue implementations.

I spent 3 years building a high scale crawler on top of Celery.

I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1].

The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them.

Celery is also extremely complex.

I would recommend https://dramatiq.io/ instead.

[1]: https://github.com/celery/celery/issues/4426

I have been using dramatiq lately (celery alternative) and so far I'm happy with it.

this is a more robust solution. https://dramatiq.io/

Dramatiq -> Just another celery?

Other options are DjangoQ and Huey, which tend to work ok. Of the two I prefer DjangoQ. Database backed, don't require the Redis/Celery rigmarole.

If you want a "simple" solution for sending email, the simplest that will likely last you the longest is using DjangoQ to create a background task that looks for model records that have not been emailed. Then use Sendgrid with an email backend to send them. You can use a library that already provides a Sendgrid email backend as well.

Take a look at Django-Q I was using it before moving to celery. Seems great just wanted experience with celery. Much simpler to get setup. Even use Django admin to schedule your tasks. Other option is a management command thats called using the full path of the python virtual env from a cron entry.

App Platform is a great concept, but we hit a dealbreaking road block when trying to migrate some Python apps with job queues. Their runtime (gVisor) doesn't support semaphore locks, which is used by Pythons multiprocessing and in turn used by most job runners (we discovered it with django-q, but I think most, if not all of them including Celery, rely on this, see link below).

The build times for Dockerfiles are also atricious… our build failed after 40 minutes by running out of memory and the multi-stage Dockerfile really wasn't anything special. We would have just used the images hosted on Github Container Registry, but App Platform only supports a limited range of Docker registries too. Note: the images build in 3 minutes on Github Actions.

As far as I can see it is also not possible to add any block storage too. While I mostly work on projects that use object storage anyway, SOME things just need persistent block storage. Which is annoying, since DigitalOcean HAS block storage… just not for App Platform.

I really wanted to use it, but man they make it hard.

https://github.com/Koed00/django-q/issues/522#issuecomment-1...

I use Django Q with ORM broker. Store tasks in db and retry if failed. You can also view/manage your queue in Django admin if you use ORM as broker (https://django-q.readthedocs.io/en/latest/brokers.html#django-orm).

django-q - Light weight task queue. When celery is too much over head.

Check out https://github.com/Koed00/django-q