Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Python Queue Projects
-
-
boltons
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
-
django-q2
A multiprocessing distributed task queue for Django. Django Q2 is a fork of Django Q. Big thanks to Ilan Steemers for starting this project. Unfortunately, development has stalled since June 2021. Django Q2 is the new updated version of Django Q, with dependencies updates, docs updates and several bug fixes. Original repository: https://github.com/Koed00/django-q
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
That's pretty cool. Reckon it would work with existing code that calls Redis over the wire for RQ?
https://python-rq.org
Project mention: Boltons is a set of over 250 BSD-licensed, pure-Python utilities | news.ycombinator.com | 2023-12-11
I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is created by docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.
____
¹ https://github.com/coleifer/huey/
Project mention: Ask HN: High quality Python scripts or small libraries to learn from | news.ycombinator.com | 2024-04-19I've recently looked at tasktiger https://github.com/closeio/tasktiger. It's a simple queue system that helped me understand how workers and schedulers work.
Project mention: Running Procrastinate from command line throwing exception | /r/learnpython | 2023-06-05I did find this PR which adds a much more detailed description of what to do, although some of it is a bit outdated.
Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08
Project mention: Litestar – powerful, flexible, and highly performant Python ASGI framework | news.ycombinator.com | 2024-02-26
To make sure you that the message you are trying to retrieve hasn't been locked already by another worker.
[0]: https://github.com/litements/litequeue/
[1]: https://github.com/litements/litequeue/blob/3fece7aa9e9a31e4...
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Queue related posts
- Choose Postgres Queue Technology
- Nextflow: Data-Driven Computational Pipelines
- Running Procrastinate from command line throwing exception
- Background jobs with Django
- The Many Problems with Celery
- Keep the Monolith, but Split the Workloads
- Launch HN: Sidekick (YC W23) – AI support bot for developers
-
A note from our sponsor - InfluxDB
www.influxdata.com | 24 Apr 2024
Index
What are some of the best open-source Queue projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | rq | 9,503 |
| 2 | boltons | 6,415 |
| 3 | huey | 4,877 |
| 4 | kombu | 2,748 |
| 5 | arq | 1,904 |
| 6 | tasktiger | 1,356 |
| 7 | Streamz | 1,217 |
| 8 | mrq | 876 |
| 9 | procrastinate | 730 |
| 10 | django-background-tasks | 584 |
| 11 | KQ | 567 |
| 12 | wakaq | 565 |
| 13 | saq | 372 |
| 14 | django_dramatiq | 324 |
| 15 | django-q2 | 279 |
| 16 | Flask-RQ2 | 223 |
| 17 | think-async | 222 |
| 18 | rele | 199 |
| 19 | simpleq | 165 |
| 20 | youtube-dl-nas | 151 |
| 21 | litequeue | 137 |
| 22 | HDR-Multi-Tool | 70 |
| 23 | spinach | 61 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com