Our great sponsors
-
spider
spider is an OD crawler that crawls through opendirectories and indexes the urls (by pyDiablo)
-
ODmovieindexer
Extract and index movie information of movies found in open directories posted on r/opendirectories.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
odcrawler-scanner
A reddit bot that scans ODs over at /r/OpenDirectories and submits the results to the ODCrawler discovery server
-
DiskCache
Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
If any of you is willing to help, I've just uploaded the code to Github. I've added as many comments as I can to help you understand the code.
For my indexer (https://github.com/LaundroMat/ODmovieindexer) I tried crawling by myself too, but I gave up because there were too many special cases to take into account. I used the text files generated by ODScanner as a basis for the URL's to index.
I also wrote a NodeJS wrapper for ODD (https://github.com/Chaphasilor/open-directory-downloader) so that I could easily use ODD in my other projects, you might wanna do the same with Python? This way everyone who knows Python could make use of ODDs edge-case handling and stability!
This way you can also evolve your application to become async. As your using requests rather than aiohttp, may I suggest you to use gevent with a pool of requests in parallel (not too much ~ 10). You can look at this file as an example.
See: https://github.com/KoalaBear84/OpenDirectoryDownloader/tree/master/OpenDirectoryDownloader.Tests/Samples
Do you know this project which covers most your needs ? http://www.grantjenks.com/docs/diskcache/