imsy
cdc-file-transfer
| imsy | cdc-file-transfer | |
|---|---|---|
| 1 | 30 | |
| 10 | 2,978 | |
| - | - | |
| 10.0 | 0.0 | |
| almost 4 years ago | almost 2 years ago | |
| Go | C++ | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
imsy
-
CDC File Transfer
I did a similar thing a few years ago
https://github.com/mkmik/imsy
It's less sophisticated but it uses the same core idea and the implementation is super simple
cdc-file-transfer
-
Parsync, a tool for parallel SSH transfers – 7x faster than rsync
This is less of a usable tool and more of a concept right now, but there are algorithmic ways to do better than rsync.
https://github.com/google/cdc-file-transfer
Hint: I really like the animated gifs on that page but they are best viewed frame-by-frame like a presentation.
-
The rsync algorithm (1996) [pdf]
Rsync is one of my favorite programs. I use it daily. The CLI is a bit quirky (e.g. trailing slashes), but once you get used to it, it makes sense. And I really always use the same flags: `-avmLP`, with `-n` for dry runs.
One alternative I'd like to try is Google's abandoned CDC[1], which claims to be up to 30x faster than rsync in certain scenarios. Does anyone know if there is a maintained fork with full Linux support?
[1]: https://github.com/google/cdc-file-transfer
-
CDC File Transfer
You can find some thoughts on it not working for Linux to Linux, and more broad compatibility, here[1] and here[2].
[1] - https://github.com/google/cdc-file-transfer/issues/56#issuec...
-
Do you have any experience with cdc_rsync?
google/cdc-file-transfer: Tools for synching and streaming files from Windows to Linux (github.com)
-
Is it possible, and if so, is it common, to insert arbitrary bytes in a file?
I have been reading about contend defined chunking. If you look at Google's cdc-transfer and other tools that use content-defined-chunking (e.g. restic, kopia, borg), they talk about how great CDC is for insertions that would otherwise shift the boundaries.
-
Google Says it is Still -Committed to Games as an Industry-
While I am no fan of Google or Stadia, this is wrong. They open sourced the CDC file transfer software they used for syncing files from Windows to Linux.
- CDC File Transfer: "Born from the ashes of Stadia, this repository contains tools for syncing and streaming files from Windows to Linux. They are based on Content Defined Chunking (CDC), in particular FastCDC, to split up files into chunks."
- Born from the ashes of Stadia, this repository contains tools for synching and streaming files from Windows to Linux.
What are some alternatives?
pgcapture - A scalable Netflix DBLog implementation for PostgreSQL
got - Version control, like Git, but for files and directories of any size. E2E Encrypted
tiflow - This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
bita - Differential file synchronization over http
d2 - D2 is a modern diagram scripting language that turns text to diagrams.