scaling-to-distributed-crawling
Redis
Our great sponsors
| scaling-to-distributed-crawling | Redis | |
|---|---|---|
| 5 | 318 | |
| 36 | 64,821 | |
| - | 2.1% | |
| 0.0 | 9.7 | |
| over 2 years ago | 4 days ago | |
| HTML | C | |
| MIT License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
scaling-to-distributed-crawling
-
DOs and DON'Ts of Web Scraping
We published a repository and blog post about distributed crawling in Python. It is a bit more complicated than what we've seen so far. It uses external software (Celery for asynchronous task queue and Redis as the database).
- Mastering Web Scraping in Python: Scaling to Distributed Crawling - ZenRows
- Mastering Web Scraping in Python: Scaling to Distributed Crawling – ZenRows
-
Mastering Web Scraping in Python: Scaling to Distributed Crawling
We will start to separate concepts before the project grows. We already have two files: tasks.py and main.py. We will create another two to host crawler-related functions (crawler.py) and database access (repo.py). Please look at the snippet below for the repo file, it is not complete, but you get the idea. There is a GitHub repository with the final content in case you want to check it.
Redis
-
Valkey Is Rapidly Overtaking Redis
One of the challenges Redis labs here have is that there's very little reason for their userbase to stay loyal to them.
antirez retired from Redis development a few years ago.
From https://github.com/redis/redis/graphs/contributors it looks like activity since he left has been mostly from people who didn't overlap with him much.
Redis Labs have not shown themselves to be outstanding stewards of the project as far as I can tell. Why shouldn't people support the fork?
-
Handling Multiple requests with Redis and Bullmq
Redis
- Redis is not "open core" (2021)
-
Software Engineering Workflow
Redis - real time data storage with different data structures in a cache
-
Redict 7.3.0, a copyleft fork of Redis, is now available
[0] https://github.com/redis/redis/blob/unstable/CONTRIBUTING.md
- It has been ten days since the last commit was pushed to Redis
-
Containerize your multi-services app with docker compose
Cache: a Redis cache
- Fix Redis Drama
- Redis changes license from BSD-3 to dual RSALv2+SSPLv1
- Change license from BSD-3 to dual RSALv2+SSPLv1
What are some alternatives?
celery - Distributed Task Queue (development branch)
Redis - 🚀 A robust, performance-focused, and full-featured Redis client for Node.js.
colly - Elegant Scraper and Crawler Framework for Golang
LevelDB - LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
newspaper - newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Polly - Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.
PeARS-orchard - This is the development version of PeARS, the people's search engine. More compact but less robust than PeARS-lite. If you just want to use PeARS as a local indexer, use PeARS-lite instead.
storm-crawler - A scalable, mature and versatile web crawler based on Apache Storm
Riak - Riak is a decentralized datastore from Basho Technologies.