Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

• bitmagnet

  7 1,904 9.1 Go

  A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.

  

Appreciate that you took the time to file an issue:

https://github.com/bitmagnet-io/bitmagnet/issues/11

• shiny-adventure

  1 0 10.0 Python

  Discontinued Torrent database project. The torrent database is populated by crawling the DHT network. Old POC Project.

> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature.

Heh. That was one of my first projects when I was still learning to code back in 2012.

https://github.com/laino/shiny-adventure

The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD:

https://github.com/laino/DHT-Torrent-database-Worker

The code is abhorrent and you absoltely shouldn't use it, but it worked.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• DHT-Torrent-database-Worker

  1 15 3.8 Python

  I was learning to code. No bully. Old project built sometime in 2012. Crawls the DHT network, downloads torrent metadata from peers and uploads that torrent information into a database.

> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature.

Heh. That was one of my first projects when I was still learning to code back in 2012.

https://github.com/laino/shiny-adventure

The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD:

https://github.com/laino/DHT-Torrent-database-Worker

The code is abhorrent and you absoltely shouldn't use it, but it worked.

• rarbg-selfhosted

  7 131 10.0 Go

  Discontinued A self-hosted Torznab API for the RARBG backup, compatible with Prowlarr, Radarr, Sonarr etc.

And the predecessor to this: https://github.com/mgdigital/rarbg-selfhosted

It's now archived due to effort being redirected to Bitmagnet.

• mldht

  2 144 10.0 Java

  Bittorrent Mainline DHT implementation in java

Oh, I already have implemented something[0] like that. It doesn't need lambdas or anything "cloud scale" like that. You "just" need a few dozen to a hundred IP addresses assigned to one machine and run a multi-homed DHT node on that to passively observe traffic from multiple points in the keyspace.

But neither of these approaches is what I'd call a "super simple DHT crawler CLI tool" that the initial comment was asking about. BEP51 is intended to make crawling simple enough that it can run on a single home internet connection, but a proper implementation still isn't trivial.

[0] https://github.com/the8472/mldht

• Magnetico

  14 2,665 1.7 Go

  Discontinued Autonomous (self-hosted) BitTorrent DHT search engine suite.

This is really neat. I'll need to check it out. A couple years ago I ran my own instance of Magnetico (https://github.com/boramalper/magnetico), but this project looks a lot more polished.

• dhtcrawler2

  1 993 0.0 Batchfile

  dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword (by btdig)

  

Isn't this much the same as btdig.com which is based on: https://github.com/btdig/dhtcrawler2

I use this service to do security research a fair bit. It'd be nice if there was a higher quality self-hosted version I could use so I'll be watching this project with interest!

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• content-seeder

  2 2 10.0 Java

I wrote a project to do this using libtorrent a while ago, but unfortunately libtorrent crashes when seeding thousands of torrents at once, which is the current blocker. I haven't been working on it since.

https://github.com/chhs1/content-seeder

Suggest a related project