odcrawler-scanner
wget2
| odcrawler-scanner | wget2 | |
|---|---|---|
| 10 | 11 | |
| 4 | - | |
| - | - | |
| 0.0 | - | |
| over 1 year ago | - | |
| JavaScript | ||
| GNU General Public License v3.0 only | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
odcrawler-scanner
-
A few tips for the newcomers on this sub !
ODCrawler: With it, as a bonus, you can download their database. It's an opensource project. Your contributions (manpower and financial) are welcome.
-
Open Directory Finder
Open Directory Crawler https://odcrawler.xyz
-
How to get the list of the working ODs released this sub ?
Visit this site: ODCrawler. It's a search engine focused on ODs and every links posted here are indexed
-
Indexed DDL's
So it's a curated version of ODCrawler?
-
08/03/2021 Daily Post
https://odcrawler.xyz/ should have these when ODScanner scanned them
-
Just a few questions about index, parent directories, etc. (Newb)
And then there's a "fourth" way, which actually isn't for finding ODs, but the files within them (because that's often what you're actually after). It's called ODCrawler and it's a search engine for finding links from those ODs that link to the files. So if you're looking for a specific file where you roughly know the name of it, it's a good idea to try using ODCrawler! (full disclosure, I'm one of the guys working on ODCrawler, so I'm biased ^^)
-
Are "Index of" search results diminishing?
Shameless plug once again, but you could use https://odcrawler.xyz instead of a direct google search.
-
Why infinite scroll?
The URL is https://odcrawler.xyz
-
CALISHOT 2021-03: Find ebooks among 453 Calibre sites
odcrawler which is permanently indexing ODs content.
- Need help with an OD indexer that I am writing in Python
wget2
-
A few tips for the newcomers on this sub !
Concurrent downloadsAble to preserve the original treeClient/Server modeCLITUIGUIWeb UIBrowser pluginwgetNYNY??Y?wget2YYNY????aria2YNYYY?Y?rcloneYYNY??Y?IDMYNNNNYNNJDownloader2YNYNNYNN
- The curl-wget Venn diagram
- GNU Wget2 now has semi-experimental DANE support!
- libwget forms the core of wget2, an HTTP(S) file downloader and website mirror utility, written entirely in C, and stuffed full of modern features.
-
Which is more common out of the box, curl or wget?
A libwget is one of the goals of wget2.
-
Command-line Tools can be 235x Faster than your Hadoop Cluster
Another tip is using wget2 instead of wget if you're mirroring a site (but this is more I/O tip than computationally heavy) https://gitlab.com/gnuwget/wget2/-/wikis/home
Sadly, wget2 doesn't support WARC last time I checked, but wget2 comes with a `--max-threads` parameter that together with `--mirror` and `--tries` makes it trivial to mirror even the slowest websites out there.
-
wget2 is a dead project or isn’t it?
Yup, there are some other contributors of course, but by and large, it's just Tim aka rockdaboot: https://gitlab.com/gnuwget/wget2/-/commits/master
-
I need to download 50 files in parallel chunks using wget.
wget 2.0 alpha supports multiple parallel downloads, if you don't mind installing another program for what I assume is a one-off task.
-
Is there a command "wget2" ?
There's a link to https://gitlab.com/gnuwget/wget2 on the wget homepage.
-
Where to extract and install system type files?
My package manager, Synaptic, does not have wget2 so I am getting it from gitlab:' https://gitlab.com/gnuwget/wget2/-/blob/master/README.md
What are some alternatives?
open-directory-downloader - A NodeJS wrapper around KoalaBear84/OpenDirectoryDownloader
axel - Lightweight CLI download accelerator
calishot
aria2 - aria2 is a lightweight multi-protocol & multi-source, cross platform download utility operated in command-line. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink.
ODmovieindexer - Extract and index movie information of movies found in open directories posted on r/opendirectories.
ht - Friendly and fast tool for sending HTTP requests
local-aircraft-reddit-bot - Local Aircraft Reddit Bot
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
spider - spider is an OD crawler that crawls through opendirectories and indexes the urls
fd - A simple, fast and user-friendly alternative to 'find'
ODBParser - OSINT tool to search, parse and dump only the open Elasticsearch and MongoDB directories that have the data you care about exposing
wgetmicro