Top 23 task-queue Open-Source Projects

• rq

  27 9,518 8.3 Python

  Simple job queues for Python

  

Project mention: Redis Re-Implemented with SQLite | news.ycombinator.com | 2024-04-14

That's pretty cool. Reckon it would work with existing code that calls Redis over the wire for RQ?

  https://python-rq.org

• Asynq

  32 8,615 7.0 Go

  Simple, reliable, and efficient distributed task queue in Go

Project mention: Looking for a language agnostic Go task queue with Redis | /r/golang | 2023-09-12

EDIT : I think the best for my case will be to use the asynq library with a custom js client, it's quite easy to implement (Detailed here)

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• flower

  3 6,169 7.7 Python

  Real-time monitor and web admin for Celery distributed task queue

• huey

  10 4,890 6.6 Python

  a little task queue for python

Project mention: Nextflow: Data-Driven Computational Pipelines | news.ycombinator.com | 2023-08-10

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is created by docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

____

¹ https://github.com/coleifer/huey/

• django-rq

  15 1,766 7.1 Python

  A simple app that provides django integration for RQ (Redis Queue)

  

Project mention: Background jobs with Django | /r/django | 2023-05-30

For simple stuff, I tend to use https://github.com/rq/django-rq. Although scheduling tasks there does not work well in my experience. If I need something to run by schedule, than better just plain cron.

• taskq

  7 1,200 0.0 Go

  Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends

• lmstfy

  1 755 5.4 Go

  A task queue with REST API

  

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• procrastinate

  7 735 9.6 Python

  PostgreSQL-based Task Queue for Python

  

Project mention: Running Procrastinate from command line throwing exception | /r/learnpython | 2023-06-05

I did find this PR which adds a much more detailed description of what to do, although some of it is a bit outdated.

• wakaq

  3 565 8.7 Python

  Background task queue for Python backed by Redis, a super minimal Celery

  

Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08

• tork

  4 467 9.7 Go

  A distributed workflow engine

Project mention: Show HN: A distributed workflow engine written in Go | news.ycombinator.com | 2023-10-01

• django-q2

  1 279 7.0 Python

  A multiprocessing distributed task queue for Django. Django Q2 is a fork of Django Q. Big thanks to Ilan Steemers for starting this project. Unfortunately, development has stalled since June 2021. Django Q2 is the new updated version of Django Q, with dependencies updates, docs updates and several bug fixes. Original repository: https://github.com/Koed00/django-q

  

• neoq

  5 243 8.3 Go

  Queue-agnostic background job library for Go, with a pleasant API and powerful features.

Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08

• rqmonitor

  1 185 0.0 Python

  Flask based more dynamic and actionable frontend dashboard for monitoring Redis Queue 👩🏿‍💻 http://python-rq.org

Project mention: Idea's for creating tools for using redis! | /r/Database | 2023-06-29

• cadence-java-client

  1 139 6.9 Java

  Java framework for Cadence Workflow Service

  

• ratus

  12 101 8.3 Go

  Ratus is a RESTful asynchronous task queue server. It translated concepts of distributed task queues into a set of resources that conform to REST principles and provides a consistent HTTP API for various backends.

  

• nq

  3 91 10.0 Go

  Cancellable, Efficient and Reliable Distributed Task Queue in Go (by DumbMachine)

• cppq

  17 90 10.0 C++

  Simple, reliable & efficient distributed task queue for C++17

• Celery-Kubernetes-Operator

  1 79 3.6 Python

  An operator to manage celery clusters on Kubernetes (Work in Progress)

  

• asyncjobs

  1 71 2.5 Go

  Asynchronous Job Processor

  

• gohive

  0 51 6.6 Go

  🐝 A Highly Performant and easy to use goroutine pool for Go

• Conveyor MQ

  2 50 0.0 TypeScript

  A fast, robust and extensible distributed task/job queue for Node.js, powered by Redis.

  

• Chunk

  1 37 0.0 PHP

  🐺 Asynchronous Task Queue Based on Distributed Message Passing for PHP.

• SmartPipeline

  1 22 7.5 Python

  A framework for rapid development of robust data pipelines following a simple design pattern

Project mention: Show HN: SmartPipeline, robust and light data pipelines in Python | news.ycombinator.com | 2023-05-03

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

task-queue related posts

• Show HN: Hatchet – Open-source distributed task queue
  22 projects | news.ycombinator.com | 8 Mar 2024
• Looking for a language agnostic Go task queue with Redis
  1 project | /r/golang | 12 Sep 2023
• I have a web app that requires allowing users to schedule a recurring task at their own specified time/interval. How would you go about it?
  2 projects | /r/golang | 6 Jul 2023
• Running Procrastinate from command line throwing exception
  1 project | /r/learnpython | 5 Jun 2023
• Background jobs with Django
  4 projects | /r/django | 30 May 2023
• The Many Problems with Celery
  5 projects | /r/Python | 22 May 2023
• Recommendation for a selfhosted worker/task queue runner?
  1 project | /r/selfhosted | 16 May 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 29 Apr 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com