For those running Go in production at scale, what do you use for distributed task queues?

This page summarizes the projects mentioned and recommended in the original post on /r/golang

Our great sponsors
  • SonarQube - Static code analysis for 29 languages.
  • InfluxDB - Collect and Analyze Billions of Data Points in Real Time
  • Mergify - Updating dependencies is time-consuming.
  • work

    Process background jobs in Go

    One of the gocraft/work maintainers here. Gocraft/work is not abandoned; we are just thinking what v2 could become, and working on v2:

  • nsq

    A realtime distributed messaging platform

    I've been using NSQ in production for a number of years now. It's also written in Go and has a first class client library maintained by the devs

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • machinery

    Machinery is an asynchronous task queue/job queue based on distributed message passing.

    Machinery ( seems to be the most popular out of the box solution. I've played with this in the past and it's worked pretty well but I'm not sure how it handles under serious volume.

  • Asynq

    Simple, reliable, and efficient distributed task queue in Go

    Asynq ( Looks promising but not quite at v1.0 yet.

  • gocelery

    Celery Distributed Task Queue in Go

    go-celery ( Celery port in Go. Not much recent activity so a bit worried it's abandoned.

  • cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    We use cadence

  • gq

    gq is a lightweight scalable message queue backed by the most popular SQL DBs. Add message queue semantics to your application without integrating extra infrastructure.

    If you want something simple

  • InfluxDB

    Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  • rabbit-hole

    RabbitMQ HTTP API client in Go

    We use Rabbitmq with streadway for the client and for some management and monitoring tasks.

  • temporal

    Temporal service

    Use ScyllaDB if you need fast inserts but eventual consistency (just remember to set larger batch_size_fail_threshold) or CockroachDB for strong consistency. I assure you, the hardware requirements will be much smaller then with Cassandra.

  • miniqueue

    A simple, single binary, message queue. Supports HTTP/2 and Redis Protocol.

    Have been working on, intentionally non distributed though for simplicity

  • docker-compose

    Temporal docker-compose files (by temporalio)

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts