SaaSHub helps you find the best software and product alternatives Learn more →
Top 3 Python crawl Projects
-
InfoSpider
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰,旨在安全快捷的帮助用户拿回自己的数据,工具代码开源,流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
-
CodeRabbit
CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
-
grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Project mention: ArchiveBox is evolving: the future of self-hosted internet archives | news.ycombinator.com | 2024-10-16https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives, put them in object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred).
-
Python crawl discussion
Python crawl related posts
-
We're losing our digital history. Can the Internet Archive save it?
-
How to download a copy of a website using Wget
-
struggling to download websites
-
Internet Archive Down, will be up and running soon (i hope).
-
best tool for downloading forum posts in real-time?
-
Best way to back up entire website on a schedule
-
Help building or mirroring docs.microsoft.com
-
A note from our sponsor - SaaSHub
www.saashub.com | 14 Feb 2025
Index
What are some of the best open-source crawl projects in Python? This list will help you:
# | Project | Stars |
---|---|---|
1 | InfoSpider | 7,919 |
2 | grab-site | 1,436 |
3 | stweet | 593 |