How to plan and execute a large number of tasks in Go?

1. Asynq

   1 34 11,405 7.8 Go

   Simple, reliable, and efficient distributed task queue in Go

   

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. gue

   2 2 291 8.0 Go

   Golang queue on top of PostgreSQL

   

   I then moved to using rabbitmq in my other projects, but if you use postgres as a DB, you can check this -> https://github.com/vgarvardt/gue

4. amqp

   3 6 4,928 1.9 Go

   Go client for AMQP 0.9.1

   

   For rabbitmq I used -> https://github.com/streadway/amqp which is deprecated now for -> https://github.com/rabbitmq/amqp091-go

5. amqp091-go

   4 3 1,783 7.7 Go

   An AMQP 0-9-1 Go client maintained by the RabbitMQ team. Originally by @streadway: `streadway/amqp`

   

   For rabbitmq I used -> https://github.com/streadway/amqp which is deprecated now for -> https://github.com/rabbitmq/amqp091-go

6. taskq

   5 7 1,276 0.0 Go

   Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends

   

   Oh my god, I have to deal with a similar system at work and I wish we hadn't tried to handroll this. If its possible you should look into something off-the-shelf. Could Celery work for 1000*10000 tasks? But if you must do it yourself, https://github.com/vmihailenco/taskq looks good

7. gocelery

   6 6 2,430 0.0 Go

   Celery Distributed Task Queue in Go

   

Suggest a related project