Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
oxen-release
Lightning fast data version control system for structured and unstructured machine learning datasets. We aim to make versioning datasets as easy as versioning code.
So on average there are 256 SHA256s per key, but worst case it takes thousands of iterations. Coding this in a GPU-friendly way was non-trivial, and there are still some GPU cycles wasted.
Disclaimer: I worked on that research, but I'm not an employe anymore.
[0]: https://github.com/CERT-Polska/phobos-cuda-decryptor-poc
Agreed. I'll shill rsync.net (no affiliation, just a happy customer) and their ZFS VM backup service. It's basically just a lightweight freebsd VM with a big ZFS volume attached, so you can `zfs send` incremental backups to it, and they support meta-snapshotting of your backup machine on their end. I wrote https://github.com/wyager/zfs-backup to manage my automatic incremental backups, and there are a number of other tools like this.
We've been working on some open source tooling called "oxen" that was built for large datasets of images, video, audio, text etc. We wanted to solve the exact problem you're flagging here with git.
Feel free to check it out here https://github.com/Oxen-AI/oxen-release#-oxen would love any feedback!