What is your favourite task queuing framework?

• celery

  43 23,498 9.5 Python

  Distributed Task Queue (development branch)

  

same! the issue has been open with a lot of people experiencing it, but no resolution https://github.com/celery/celery/issues/4185

• nifi

  35 4,381 9.9 Java

  Apache NiFi

  

Apache NiFi -> More for data analysis/transformation

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• huey

  10 4,890 6.6 Python

  a little task queue for python

Huey -> Same again?

• dramatiq

  15 4,061 7.3 Python

  A fast and reliable background task processing library for Python 3.

Dramatiq -> Just another celery?

• rq

  27 9,518 8.3 Python

  Simple job queues for Python

  

RQ -> ...?

• ApacheKafka

  104 28 0.0

  A curated re-sources list for awesome Apache Kafka

Kafka is a bit of a sledgehammer to crack a nut. It's designed to be massively scaleable and fault-tolerant, and that comes at the expense of having to run two clusters of three(ish) nodes each, where one set is the queue itself and the other set is orchestrator nodes. "Big operational problem" means that it's a lot to set up and maintain.

• BeanstalkD

  14 6,472 0.0 C

  Beanstalk is a simple, fast work queue.

  

I tried many of the options you can find here: https://taskqueues.com/. Ultimately I began using beanstalkd, and have been satisfied with it for a couple of years now. It's simple, fast, reliable, and has the basic queuing features that I need, like task handling (reserve, free), and TTL.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project