odcrawler-scanner
calishot
odcrawler-scanner | calishot | |
---|---|---|
10 | 3 | |
4 | 16 | |
- | - | |
0.0 | 0.0 | |
over 1 year ago | about 2 years ago | |
JavaScript | Python | |
GNU General Public License v3.0 only | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
odcrawler-scanner
-
A few tips for the newcomers on this sub !
ODCrawler: With it, as a bonus, you can download their database. It's an opensource project. Your contributions (manpower and financial) are welcome.
-
Open Directory Finder
Open Directory Crawler https://odcrawler.xyz
-
How to get the list of the working ODs released this sub ?
Visit this site: ODCrawler. It's a search engine focused on ODs and every links posted here are indexed
-
Indexed DDL's
So it's a curated version of ODCrawler?
-
08/03/2021 Daily Post
https://odcrawler.xyz/ should have these when ODScanner scanned them
-
Just a few questions about index, parent directories, etc. (Newb)
And then there's a "fourth" way, which actually isn't for finding ODs, but the files within them (because that's often what you're actually after). It's called ODCrawler and it's a search engine for finding links from those ODs that link to the files. So if you're looking for a specific file where you roughly know the name of it, it's a good idea to try using ODCrawler! (full disclosure, I'm one of the guys working on ODCrawler, so I'm biased ^^)
-
Are "Index of" search results diminishing?
Shameless plug once again, but you could use https://odcrawler.xyz instead of a direct google search.
-
Why infinite scroll?
The URL is https://odcrawler.xyz
-
CALISHOT 2021-03: Find ebooks among 453 Calibre sites
odcrawler which is permanently indexing ODs content.
- Need help with an OD indexer that I am writing in Python
calishot
-
CALISHOT 2022-01: Find ebooks among 373 Calibre sites this month
Other resources will be proposed on it soon, like a wiki, tips, the datasets, original calibres, and some news about related tools like calisuck, calishot ... which are now turning into a single new project and will be released soon.
-
CALISHOT 2021-06: Find ebooks among 383 Calibre sites
If you want to build the db with your own list of servers, here is the python project on Github with commands to run on you own list of servers.
-
Need help with an OD indexer that I am writing in Python
This way you can also evolve your application to become async. As your using requests rather than aiohttp, may I suggest you to use gevent with a pool of requests in parallel (not too much ~ 10). You can look at this file as an example.
What are some alternatives?
open-directory-downloader - A NodeJS wrapper around KoalaBear84/OpenDirectoryDownloader
ODmovieindexer - Extract and index movie information of movies found in open directories posted on r/opendirectories.
demeter - Demeter is a tool for scraping the calibre web ui
local-aircraft-reddit-bot - Local Aircraft Reddit Bot
spider - scripts and baselines for Spider: Yale complex and cross-domain semantic parsing and text-to-SQL challenge
spider - spider is an OD crawler that crawls through opendirectories and indexes the urls
ODBParser - OSINT tool to search, parse and dump only the open Elasticsearch and MongoDB directories that have the data you care about exposing
DiskCache - Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.
CloudScraper - CloudScraper: Tool to enumerate targets in search of cloud resources. S3 Buckets, Azure Blobs, Digital Ocean Storage Space.