For those running Go in production at scale, what do you use for distributed task queues?

This page summarizes the projects mentioned and recommended in the original post on /r/golang

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • work

    Process background jobs in Go

    One of the gocraft/work maintainers here. Gocraft/work is not abandoned; we are just thinking what v2 could become, and working on v2: https://github.com/gocraft/work/issues/120#issuecomment-530615442

  • nsq

    A realtime distributed messaging platform

    I've been using NSQ in production for a number of years now. It's also written in Go and has a first class client library maintained by the devs

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • machinery

    Machinery is an asynchronous task queue/job queue based on distributed message passing.

    Machinery (https://github.com/RichardKnop/machinery) seems to be the most popular out of the box solution. I've played with this in the past and it's worked pretty well but I'm not sure how it handles under serious volume.

  • Asynq

    Simple, reliable, and efficient distributed task queue in Go

    Asynq (https://github.com/hibiken/asynq): Looks promising but not quite at v1.0 yet.

  • gocelery

    Celery Distributed Task Queue in Go

    go-celery (https://github.com/gocelery/gocelery): Celery port in Go. Not much recent activity so a bit worried it's abandoned.

  • cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    We use cadence https://github.com/uber/cadence

  • gq

    gq is a lightweight scalable message queue backed by the most popular SQL DBs. Add message queue semantics to your application without integrating extra infrastructure.

    If you want something simple https://github.com/mattbonnell/gq

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • rabbit-hole

    RabbitMQ HTTP API client in Go

    We use Rabbitmq with streadway for the client and https://github.com/michaelklishin/rabbit-hole for some management and monitoring tasks.

  • temporal

    Temporal service

    Use ScyllaDB if you need fast inserts but eventual consistency (just remember to set larger batch_size_fail_threshold) or CockroachDB for strong consistency. I assure you, the hardware requirements will be much smaller then with Cassandra.

  • miniqueue

    A simple, single binary, message queue. Supports HTTP/2 and Redis Protocol.

    Have been working on https://github.com/tomarrell/miniqueue, intentionally non distributed though for simplicity

  • docker-compose

    Temporal docker-compose files (by temporalio)

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts