Mindcastle.io Alternatives

Similar projects and alternatives to mindcastle.io

rclone

963 43,840 9.8 Go mindcastle.io VS rclone

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
s3fs-fuse

57 8,079 8.8 C++ mindcastle.io VS s3fs-fuse

FUSE-based file system backed by Amazon S3
InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
juicefs

42 9,824 9.8 Go mindcastle.io VS juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.
gcsfuse

31 1,977 9.7 Go mindcastle.io VS gcsfuse

A user-space file system for interacting with Google Cloud Storage
seafowl

11 355 9.3 Rust mindcastle.io VS seafowl

Analytical database for data-driven Web applications 🪶
mountpoint-s3

17 4,017 9.5 Rust mindcastle.io VS mountpoint-s3

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.
thumbhash

9 3,245 2.5 Swift mindcastle.io VS thumbhash

A very compact representation of an image placeholder
SaaSHub

www.saashub.com featured

SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
extfuse

2 200 2.9 C mindcastle.io VS extfuse

Extension Framework for FUSE
azure-storage-fuse-aur

2 1 6.2 Shell mindcastle.io VS azure-storage-fuse-aur

AUR package for Azure Storage Blobfuse
s3fs

7 814 7.8 Python mindcastle.io VS s3fs

S3 Filesystem
azurefs

1 70 10.0 Python mindcastle.io VS azurefs

Mount Microsoft Azure Blob Storage as local filesystem in Linux (inactive)
catfs

4 812 3.6 Rust mindcastle.io VS catfs

Cache AnyThing filesystem written in Rust

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better mindcastle.io alternative or higher similarity.

Suggest an alternative to mindcastle.io

mindcastle.io reviews and mentions

Posts with mentions or reviews of mindcastle.io. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-09-06.

Prolly Trees
1 project | news.ycombinator.com | 9 Mar 2024

I don’t know who came first, but https://github.com/jacobgorm/mindcastle.io also uses the rsync/LBFS rolling hashes trick to split the tree data into chunks. I presented the idea at Usenix Vault 2019 https://m.youtube.com/watch?v=QgOkDiP0C4c&embeds_referring_e...
Gcsfuse: A user-space file system for interacting with Google Cloud Storage
15 projects | news.ycombinator.com | 6 Sep 2023

It is not how you would want do it for a typical ML workload, where the samples have to get randomly permuted each epoch.
Instead, tar up the files in some random order, and put the tar file on a web server or bucket, then stream then in during the first epoch, while keeping track of their byte offsets in the tar file, which you cache locally, assuming ample local Flash storage. Then permute the list of offsets and use those when reading samples for the next epoch.
If you only have local HDD then you will need a more advanced data structure like the one provided by https://github.com/jacobgorm/mindcastle.io , which will allow you to write out permuted samples at close to disk sequential write bandwidth. See my talk at USENIX Vault 2019 for a full explanation, linked from https://vertigo.ai/mindcastle/