bita VS got

Built this cdc tool for software update of embedded (Linux) systems and have deployed it with good enough performance on a couple of arm CPUs; https://github.com/oll3/bita

Though main goal has been keeping data usage low rather than speed up.

Nice write up. rsync is great as an application but I found it more cumbersome to use when wanting to integrate it into my own application. There's librsync but the documentation is threadbare and it requires an rsync server to run. I found bita/bitar (https://github.com/oll3/bita) which is inspired by rsync & family. It works more like zsync which leverages HTTP Range requests so it doesn't require anything running on the server to get chunks. Works like a treat using s3/b2 storage to serve files and get incremental differential updates on the client side!

Very cool, thanks for sharing. I did a deep dive in the past into various syncing/binary diff protocols and really liked zsync. It was probably my top choice for the application I was designing but I ended up not using it. The library I did use is called bita: https://github.com/oll3/bita. It is inspired by the same family of projects as zsync. The main advantage I found with bita is that the core logic is encapsulated in a library so that you don’t only have to use the binaries but can integrate it directly into an application. I’d be curious to know if that’s in the plans for KySync.

I've not heard the term "probabilistic tree" and I've having difficulty pulling up references. I suspect it's implemented by subpackage ptree[0]. Do you have resources on what makes probabilistic trees different from hash tables?

[0] https://github.com/gotvc/got/tree/master/pkg/gotkv/ptree

Sure, Git stores data in a trie. Each file is one blob identified by hash, and directories (called trees in Git) are blobs where each line is a directory entry with a name and the hash of a file or another tree. This means that modifying an object /down/a/long/path/like/this.txt has to create copies of all the trees on the way up. The technical term for this is "write amplification", and in Git it is affected by path length among other things.

Got stores paths in a probabilistic tree (GotKV[0]). The number of nodes before you get to data will scale logarithmically with the size of the entire filesystem, not the depth of a specific object.

Then there is the issue of large files. A file in Git is always 1 blob. Syncing a large file is not easy because if you are interrupted and have to restart, you have lost all your progress. You can't verify the hash of a blob until you have the whole thing. Got has a maximum blob size, so you'll only be buffering <2MB at a time before you can verify that the blob is correct. If a transfer is interrupted, the most you'll have to repeat is one blobs worth, plus any tree nodes above that blob.

Compared to rsync, Got uses variable size chunks and a faster content defined chunking algorithm, recently featured here on HN[1]. I haven't thought about if variable vs fixed chunks is better for file transfer, but for version control, the higher chance of convergence is important. It means you have better deduplication.

[0] https://github.com/gotvc/got/tree/master/pkg/gotkv

[1] https://news.ycombinator.com/item?id=34303497

This is a very similar to one of my projects "Got".

https://github.com/gotvc/got

The algorithms it uses are superior to rsync and git in a few ways. It comes short on features, especially for software development compared to Git. The motivation is more for personal file storage.

I notice you're using Go and AGPL licensed, so you could borrow any of Got's libraries without issue. (Got is GPL licensed.) Definitely reach out in a GitHub issue.

FastCDC is the same chunking algorithm used in Got.

https://github.com/gotvc/got

Thanks for sharing RocketGit. This is the first time I've heard of it, and yes, it does look like a cool copyleft solution to self-hosted Git.

Another interesting option is Brendan Caroll's got[0], which allows sharing of repositories over INET256[1]. I'm sure there are other P2P approaches to Git, but this one just piqued my interest. Unfortunately it has a naming conflict with OpenBSD's Game of Trees[2].

[0] https://github.com/gotvc/got

[1] https://github.com/inet256/inet256

[2] https://gameoftrees.org/

I work on a project which solves a similar use case.

https://github.com/gotvc/got

Got also does E2EE encryption, but it can additionally encrypt branch names from remote servers.

I've been working on a project "Got". Which deals with the LFS problem, mentioned in the post.

https://github.com/gotvc/got

Got isn't really trying to do software version control better than Git. It's trying to make general purpose file versioning practical, with a workflow similar to Git's.

In the same space is the key-value store underlying Got: GotKV. https://github.com/gotvc/got/tree/master/pkg/gotkv

It stores encrypted blobs in any content-addressed store, and provides a copy-on-write key-value store API.