Python task-queue

Open-source Python projects categorized as task-queue

Top 11 Python task-queue Projects

  • rq

    Simple job queues for Python

    Project mention: Redis Re-Implemented with SQLite | news.ycombinator.com | 2024-04-14

    That's pretty cool. Reckon it would work with existing code that calls Redis over the wire for RQ?

      https://python-rq.org

  • flower

    Real-time monitor and web admin for Celery distributed task queue

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • huey

    a little task queue for python

    Project mention: Nextflow: Data-Driven Computational Pipelines | news.ycombinator.com | 2023-08-10

    I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is created by docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

    ____

    ¹ https://github.com/coleifer/huey/

  • django-rq

    A simple app that provides django integration for RQ (Redis Queue)

    Project mention: Background jobs with Django | /r/django | 2023-05-30

    For simple stuff, I tend to use https://github.com/rq/django-rq. Although scheduling tasks there does not work well in my experience. If I need something to run by schedule, than better just plain cron.

  • procrastinate

    PostgreSQL-based Task Queue for Python

    Project mention: Running Procrastinate from command line throwing exception | /r/learnpython | 2023-06-05

    I did find this PR which adds a much more detailed description of what to do, although some of it is a bit outdated.

  • wakaq

    Background task queue for Python backed by Redis, a super minimal Celery

    Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08
  • django-q2

    A multiprocessing distributed task queue for Django. Django Q2 is a fork of Django Q. Big thanks to Ilan Steemers for starting this project. Unfortunately, development has stalled since June 2021. Django Q2 is the new updated version of Django Q, with dependencies updates, docs updates and several bug fixes. Original repository: https://github.com/Koed00/django-q

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • rqmonitor

    Flask based more dynamic and actionable frontend dashboard for monitoring Redis Queue 👩🏿‍💻 http://python-rq.org

    Project mention: Idea's for creating tools for using redis! | /r/Database | 2023-06-29
  • Celery-Kubernetes-Operator

    An operator to manage celery clusters on Kubernetes (Work in Progress)

  • SmartPipeline

    A framework for rapid development of robust data pipelines following a simple design pattern

    Project mention: Show HN: SmartPipeline, robust and light data pipelines in Python | news.ycombinator.com | 2023-05-03
  • django-tasks-scheduler

    Schedule async tasks using redis pub/sub.

    Project mention: django-rq-scheduler VS django-tasks-scheduler - a user suggested alternative | libhunt.com/r/django-rq-scheduler | 2023-12-15
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-04-14.

Python task-queue related posts

Index

What are some of the best open-source task-queue projects in Python? This list will help you:

Project Stars
1 rq 9,481
2 flower 6,149
3 huey 4,870
4 django-rq 1,760
5 procrastinate 725
6 wakaq 563
7 django-q2 278
8 rqmonitor 184
9 Celery-Kubernetes-Operator 78
10 SmartPipeline 21
11 django-tasks-scheduler 17
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com