For those running Go in production at scale, what do you use for distributed task queues?

• work

  5 2,403 0.0 Go

  Process background jobs in Go

  

  One of the gocraft/work maintainers here. Gocraft/work is not abandoned; we are just thinking what v2 could become, and working on v2: https://github.com/gocraft/work/issues/120#issuecomment-530615442

• InfluxDB

  www.influxdata.com featured

  Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

  

• nsq

  14 24,857 5.8 Go

  A realtime distributed messaging platform

  

  I've been using NSQ in production for a number of years now. It's also written in Go and has a first class client library maintained by the devs

• machinery

  14 7,494 4.7 Go

  Machinery is an asynchronous task queue/job queue based on distributed message passing.

  Machinery (https://github.com/RichardKnop/machinery) seems to be the most popular out of the box solution. I've played with this in the past and it's worked pretty well but I'm not sure how it handles under serious volume.

• Asynq

  32 9,396 6.1 Go

  Simple, reliable, and efficient distributed task queue in Go

  Asynq (https://github.com/hibiken/asynq): Looks promising but not quite at v1.0 yet.

• gocelery

  6 2,358 0.0 Go

  Celery Distributed Task Queue in Go

  

  go-celery (https://github.com/gocelery/gocelery): Celery port in Go. Not much recent activity so a bit worried it's abandoned.

• cadence

  20 8,182 9.8 Go

  Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

  

  We use cadence https://github.com/uber/cadence

• gq

  6 27 0.0 Go

  gq is a lightweight scalable message queue backed by the most popular SQL DBs. Add message queue semantics to your application without integrating extra infrastructure.

  If you want something simple https://github.com/mattbonnell/gq

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• rabbit-hole

  1 411 7.7 Go

  RabbitMQ HTTP API client in Go

  We use Rabbitmq with streadway for the client and https://github.com/michaelklishin/rabbit-hole for some management and monitoring tasks.

• temporal

  18 11,344 9.9 Go

  Temporal service

  

  Use ScyllaDB if you need fast inserts but eventual consistency (just remember to set larger batch_size_fail_threshold) or CockroachDB for strong consistency. I assure you, the hardware requirements will be much smaller then with Cassandra.

• miniqueue

  5 203 0.0 Go

  A simple, single binary, message queue. Supports HTTP/2 and Redis Protocol.

  Have been working on https://github.com/tomarrell/miniqueue, intentionally non distributed though for simplicity

• docker-compose

  3 302 7.1 Shell

  Temporal docker-compose files (by temporalio)

  

Suggest a related project