DiskCache VS odcrawler-scanner

python-diskcache – disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python

Based on your question about SQLite, it seems like you want to store the database inside of the program as opposed to on the internet. Furthermore, your data doesn't seem to be super relational to my knowledge. You might be better off using something like diskcache to store the data instead.

Give Grant some love and give grantjenks/python-diskcache a ⭐.

Do you know this project which covers most your needs ? http://www.grantjenks.com/docs/diskcache/

ODCrawler: With it, as a bonus, you can download their database. It's an opensource project. Your contributions (manpower and financial) are welcome.

Open Directory Crawler https://odcrawler.xyz

Visit this site: ODCrawler. It's a search engine focused on ODs and every links posted here are indexed

So it's a curated version of ODCrawler?

https://odcrawler.xyz/ should have these when ODScanner scanned them

And then there's a "fourth" way, which actually isn't for finding ODs, but the files within them (because that's often what you're actually after). It's called ODCrawler and it's a search engine for finding links from those ODs that link to the files. So if you're looking for a specific file where you roughly know the name of it, it's a good idea to try using ODCrawler! (full disclosure, I'm one of the guys working on ODCrawler, so I'm biased ^^)

Shameless plug once again, but you could use https://odcrawler.xyz instead of a direct google search.

The URL is https://odcrawler.xyz

odcrawler which is permanently indexing ODs content.