-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
It doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...
related: https://torrent-paradise.ml/ Also based on IPFS with this HTTP gateway. I find a lot of torrents in there.
>That's interesting. Are you aware of the absurd-sql project?
No, thanks for pointing that out, will take a look into it.
> Also, how do peers find each other?
The idea is to use the torrents as a common shareable resource where given the peers have the same interest in that torrent, lets say the torrents works as a "meta-database" with just enough immutable metadata info, giving the developers of that application a list of peers that will have the same RPC service you designed, with a interface that suits your purpose according to your application goals (a distributed Youtube for instance), and giving you know and designed that API yourself whatever you want from each node, lets say a piece of data, be it a file or a database key-value range, you can ask your API for it, combining the torrent peers and whatever distribution combo you need.
> And finally, do you intend to open source it?
I've just did
https://github.com/mumba-org/mumba
It's badly documented given i'm on the final touches before a proper launch, but the "documentation" of the storage layer is planned for today, giving how important it is for the whole thing.
The first thing i've tried to get right was the storage layer, and the capacity to use mutable sqlite databases over torrent (together with files which are simpler given they are meant to be immutable).
Given theres no proper doc yet, i can point out to the source at
https://github.com/mumba-org/mumba/tree/main/lib/storage
where:
https://github.com/mumba-org/mumba/tree/main/lib/storage/bac...
is a modified chrome cache storage layer (which is the real underlying disk storage) that abstract the files and databases storages that from the bit-torrent layer perspective are on the disk.
https://github.com/mumba-org/mumba/blob/main/lib/storage/sto...
is the front end
and the:
https://github.com/mumba-org/mumba/blob/main/lib/storage/tor...
is the main abstraction that may be a "fileset"(collection of files as in torrent) or a dataset/database, which in this case you can get the sqlite db handle from the torrent object and deal with it as normal database.
I've take care to enable a key-value store over the sqlite btree, so both form of databases are possible, a key-value and a normal SQL database.
Key-values are important for the distributed case where you may want the nodes to have partial data and abstract a SQL layer (or whatever) on top of the distributed nodes, which is a better solution for distributed storage.
For the database distribution, a 64k SQLite memory page maps to a torrent piece of the same size, which can be synchronized over other peers that knows what's the root of the merkle tree of the given database is (you can use the RPC layer to coordinate this or use the bit-torrent DHT updating your slot with the new merkle root)
BTW This is what i'm using to distribute the applications, the DHT which points to a "database torrent" which in turn is a index to other files and database torrents.