wdt VS wcp

I like the way you're thinking here, I think the limitations you mentioned with gemini may stand... for me it's kind of like the limitations generally speaking with markdown. Doesn't leave much room for doing stuff like parsing the raw data when they aren't in a hierarchical structure with xpaths you can target and stuff like that, it just throws out so much baby with the bathwater that I'm ready to scream infanticide.

Any thoughts on fast experimental protocols like warp data transfer [1] or fast and secure protocol [2] ? I know they're not exactly the most open things or wellsupported in terms of what you're looking for but I've been really wondering when we're going to start seeing pressure to relieve network congestion using stuff like this.

[1] https://github.com/facebookarchive/wdt

That's great stuff, I wonder how it compares to wdt[0] when there's only one TCP path. This has been my go-to tool to transfer files on the network.

0: https://github.com/facebook/wdt

I guess? If you’re building parallel infrastructure (vlans are not enough obviously) just for running Aspera over it might not be the worst thing ever, but that’s an expensive way to live and that’s before you pay for the A$pera licenses. There are free and better behaved platforms out there like https://github.com/facebook/wdt if you don’t want other applications’ TCP sessions to time out while you’re trying to squeeze out the last half percent with Aspera.

Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?)

1: https://github.com/wheybags/wcp

The chart is over NFS, but the listed speeds in the blog, and on github[1] are from a copy on a local SSD. That is a bit confusing though, maybe I should make it more clear. I used the network copy for ETA calculation because it was an easy way to make the transfer take longer -.if the whole copy is only a few seconds long it's difficult to meaningfully compare ETA estimation accuracy between two approaches. I would like to have more performance data though.

1: https://github.com/wheybags/wcp/#how-fast

Unix cp, but with a proper progress bar, and much faster:

https://github.com/wheybags/wcp

Getting close now but not ready for real use. io_uring is awesome.