rq
Celery-Kubernetes-Operator
Our great sponsors
rq | Celery-Kubernetes-Operator | |
---|---|---|
27 | 1 | |
9,518 | 79 | |
1.2% | - | |
8.3 | 3.6 | |
7 days ago | 5 months ago | |
Python | Python | |
GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rq
-
Redis Re-Implemented with SQLite
That's pretty cool. Reckon it would work with existing code that calls Redis over the wire for RQ?
https://python-rq.org
-
The Many Problems with Celery
https://github.com/rq/rq is to the rescue.
-
Keep the Monolith, but Split the Workloads
We use RQ[0], it has Redis as a dependency. It’s pretty straightforward and we’re very happy with it. If you are using Django you may want to look at Django RQ[1] as well. RQ has built in scheduling capabilities these days, but historically it did not so we used (and still use) RQ Scheduler[2] which I think still has some advantages over the built in stuff.
[0] https://python-rq.org/
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
Also had a similar experience using RabbitMQ with Django+Celery. Extremely complicated and workers/queues would just stop for no reason.
Moved to Python-RQ [1] + Redis and been rock solid for years now.
[1] https://python-rq.org/
- Ask HN: Redis Queue Hacks and Questions
- What libraries do you use the most alongside django?
-
Recommendations other than celery to send an API processing in background, which would only take 5 mins to process and API usage would be once a month or so.
Yep, rq is simple and good: https://python-rq.org/ It also has a Django wrapper: https://github.com/rq/django-rq
-
GPU instance crashes when two python processes use the same pt file
We have a GPU (G5) instance that uses Python RQ (https://python-rq.org/).
- Dynamically update periodic tasks in Celery and Django
- Celery + RabbitMQ alternatives
Celery-Kubernetes-Operator
-
Help: from Docker-Compose to Production (EC2, ECS, EKR)
The take-out from that course is: don't deploy anything stateful on Kubernetes in production, period. Even disregarding that, don't deploy anything stateful that doesn't come in a form of an operator. For celery, https://github.com/celery/Celery-Kubernetes-Operator is a WIP, so obviously not suitable for anything.
What are some alternatives?
celery - Distributed Task Queue (development branch)
flower - Real-time monitor and web admin for Celery distributed task queue
huey - a little task queue for python
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
mrq - Mr. Queue - A distributed worker task queue in Python using Redis & gevent
kopf - A Python framework to write Kubernetes operators in just a few lines of code
procrastinate - PostgreSQL-based Task Queue for Python
deploy-ecs - This project aims to build, deploy and configure your services into containers of AWS ECS
Apache Kafka - Mirror of Apache Kafka
sieve - Automatic Reliability Testing for Kubernetes Controllers and Operators