How to plan and execute a large number of tasks in Go?

• Asynq

  32 8,615 7.0 Go

  Simple, reliable, and efficient distributed task queue in Go

• gue

  2 242 8.3 Go

  Golang queue on top of PostgreSQL

I then moved to using rabbitmq in my other projects, but if you use postgres as a DB, you can check this -> https://github.com/vgarvardt/gue

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• amqp

  6 4,797 1.9 Go

  Go client for AMQP 0.9.1

For rabbitmq I used -> https://github.com/streadway/amqp which is deprecated now for -> https://github.com/rabbitmq/amqp091-go

• amqp091-go

  2 1,311 7.4 Go

  An AMQP 0-9-1 Go client maintained by the RabbitMQ team. Originally by @streadway: `streadway/amqp`

  

For rabbitmq I used -> https://github.com/streadway/amqp which is deprecated now for -> https://github.com/rabbitmq/amqp091-go

• taskq

  7 1,199 0.0 Go

  Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends

Oh my god, I have to deal with a similar system at work and I wish we hadn't tried to handroll this. If its possible you should look into something off-the-shelf. Could Celery work for 1000*10000 tasks? But if you must do it yourself, https://github.com/vmihailenco/taskq looks good

• gocelery

  6 2,316 0.0 Go

  Celery Distributed Task Queue in Go

  

Suggest a related project