SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Python Redi Projects
-
I spent 3 years building a high scale crawler on top of Celery.
I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1].
The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them.
Celery is also extremely complex.
I would recommend https://dramatiq.io/ instead.
-
Project mention: What libraries do you use the most alongside django? | reddit.com/r/django | 2023-01-02
-
Sonar
Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
Project mention: Awesome list of VCs investing in commercial open-source startups | reddit.com/r/opensource | 2022-09-14
OpenReplay - Elastic license
-
dev-setup
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
Something like this at least is the most direct answer to your question, as opposed to "you're doing it wrong" which unfortunately seems to be more upvoted. An example of something like this might be https://github.com/donnemartin/dev-setup
-
Project mention: Python Celery - high level overview(animated video) | reddit.com/r/Python | 2023-01-29
Not sure what your criteria for good observability ist, but flower served me well enough for this in the last couple of years.
-
Project mention: Ask HN: What are the best tools for web scraping in 2022? | news.ycombinator.com | 2022-08-10
11. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis.
-
Stream-Framework
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
Project mention: Create a customer-admin chat portal for online ordering website | reddit.com/r/react | 2023-01-30Have you worked with Stream before? I think it's what you're looking for.
-
InfluxDB
Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.
-
Project mention: What's the best thing you've learned about Django this year? | reddit.com/r/django | 2022-12-14
Funny, just this moment i finally switched from Celery to huey. And so far I don't regret. huey looks very promising, has good documentation and is well integrated into DJango. You should give it a try: https://github.com/coleifer/huey
-
I spent 3 years building a high scale crawler on top of Celery.
I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1].
The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them.
Celery is also extremely complex.
I would recommend https://dramatiq.io/ instead.
-
Project mention: Wrong dashboard while adding flask-admin to project | reddit.com/r/codehunter | 2022-03-14
I'm trying to extend the flask-base project https://github.com/hack4impact/flask-base/tree/master/app. This uses the the application factory pattern in app/init.py and blueprints.
-
It is feasible and there are a number of ways to do it. There are a few issues, though, mainly how to make sure that you are accessing the latest data and not getting part of the data mid write. A database handles these issues pretty well, since generally they are meant to be an intermediary between different processes. Another solution, that is often more flexible, is a message queue, like this one: https://github.com/celery/kombu
-
First, it was aioredis library. We are using sentinel based client because with this we can achieve failover easily. Aioredis spawn pool of connections, that transparently reconnects (and here third thing — FOREVER, hello DDOS) to our sentinel nodes, and then to master node. It supposed to do so. Also, we found that if you are not limiting maximum connections count, library will do it for you and set it as 2 ** 31 (here you can see it) — this is fourth thing. Furthermore, pool in our version (2.0.1) not closing automatically, and it makes the problem worse.
-
wget https://github.com/laixintao/iredis/releases/latest/download/iredis.tar.gz \ && tar -xzf iredis.tar.gz \ && ./iredis
-
-
Project mention: Recommendations other than celery to send an API processing in background, which would only take 5 mins to process and API usage would be once a month or so. | reddit.com/r/django | 2022-12-12
Yep, rq is simple and good: https://python-rq.org/ It also has a Django wrapper: https://github.com/rq/django-rq
-
-
RQ-Scheduler is another simpler alternative (rq/rq-scheduler: A lightweight library that adds job scheduling capabilities to RQ (Redis Queue) (github.com)) that appears to be good for such purposes. It's not immediately clear if it would suffer from the same issues, but it seems not (Redis manages issues with data loss well, a separate queue is used for the scheduled tasks, etc.). Is anyone aware of any drawbacks to using RQ-Scheduler for something like this?
-
nagios-plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
-
redis-py-cluster A port of redis-rb-cluster to Python. Supports majority of redis-py functionality. Is in active development.
-
-
Project mention: Web Scraping with Python: from Fundamentals to Practice | reddit.com/r/Python | 2022-06-23
For anyone who goes with requests as your HTTP client, I would highly recommend adding requests-cache for a nice performance boost.
-
Project mention: Hosting recommendation for a new e-commerce website. | reddit.com/r/webhosting | 2023-01-23
Also, WordOps
-
Project mention: How to join using Faust Streaming (Python implementation of Kafka Streams API)? | reddit.com/r/apachekafka | 2023-01-31
The forked one, https://github.com/faust-streaming/faust, has been updated but still doesn’t seem to support joins.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-01-31.
Python Redis related posts
- Sidewinder: open source Django starter kit that focuses on good defaults, developer experience, and deployment
- Django Styleguide
- RQ-Scheduler for tasks in far future?
- Is there any way for hGetAll to return a key-value pair list instead of a simple list?
- What's the best thing you've learned about Django this year?
- Question: Is there selfhosted email server with spam filtering by offline AI model?
- Recommendations other than celery to send an API processing in background, which would only take 5 mins to process and API usage would be once a month or so.
-
A note from our sponsor - #<SponsorshipServiceOld:0x00007fea59581290>
www.saashub.com | 3 Feb 2023
Index
What are some of the best open-source Redi projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | celery | 20,878 |
| 2 | rq | 8,737 |
| 3 | openreplay | 6,817 |
| 4 | dev-setup | 5,874 |
| 5 | flower | 5,545 |
| 6 | scrapy-redis | 5,230 |
| 7 | Stream-Framework | 4,678 |
| 8 | huey | 4,340 |
| 9 | dramatiq | 3,430 |
| 10 | flask-base | 2,841 |
| 11 | kombu | 2,493 |
| 12 | aioredis | 2,263 |
| 13 | iredis | 2,232 |
| 14 | Gopherus | 2,174 |
| 15 | django-rq | 1,610 |
| 16 | arq | 1,458 |
| 17 | rq-scheduler | 1,280 |
| 18 | nagios-plugins | 1,083 |
| 19 | redis-py-cluster | 1,077 |
| 20 | walrus | 1,062 |
| 21 | requests-cache | 1,060 |
| 22 | WordOps | 995 |
| 23 | faust | 845 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com