Estranged.Lfs VS dud

Did some more research to see if anything had changed in this space. I found two interesting projects (haven't used them myself yet though):

One in C# (with support for auth)

https://github.com/alanedwardes/Estranged.Lfs

One in Rust (but no Auth, have to run reverse proxy)

https://github.com/jasonwhite/rudolfs

Both seem interesting. Anyone use these?

I've used DVC in the past and generally liked its approach. That said, I wholeheartedly agree that it's clunky. It does a lot of things implicitly, which can make it hard to reason about. It was also extremely slow for medium-sized dataset (low 10s of GBs).

In response, I created a command-line tool that addresses these issues[0]. To reduce the comparison to an analogy: Dud : DVC :: Flask : Django.

[0]: https://github.com/kevin-hanselman/dud

There is also https://github.com/kevin-hanselman/dud

I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think.

[0]: https://github.com/kevin-hanselman/dud

[1]: https://kevin-hanselman.github.io/dud/benchmarks/

I don't know about replacing Make with Docker, but I use the two together to good effect. One of my favorite hacks is adding a 'docker-%' rule in my Makefile to run make commands in a Docker image[1]. It's a bit mind-bending, and there's a few gotchas, but it works surprisingly well for simple rules.

[1]: https://github.com/kevin-hanselman/dud/blob/e98de8fcdf7ad564...

Thanks for sharing your experience. It's non-trivial and surprising behavior like this that drove me to build a custom system[0] myself. When I started researching version control tools for large files, I remember feeling like git-annex and Git LFS were awkwardly bolted onto Git; Git simply wasn't designed for large files. Then I found DVC[1], and its approach rang true for me. However, after using DVC for a year or so, I grew tired of DVC's many puzzling behaviors (most of which are outlined in the README at [0]). In the end, I built the tool I wanted for the job -- one that is exceptionally simple and fast.

[0]: https://github.com/kevin-hanselman/dud