oxen-release VS dud

We've been working on a data version control system called "oxen" optimized for large unstructured datasets that we are seeing more and more with the advent of many of the generative AI techniques.

Many of these datasets have many many images, videos, audio files, text as well as structured tabular datasets that git or git-lfs just falls flat on.

Would love anyone to kick the tires on it and let us know what you think:

https://github.com/Oxen-AI/oxen-release

The commands are mirrored after git so it is easy to learn, but optimized under the hood for larger datasets.

Super cool! Would love to see an integration with Oxen and their data version control https://github.com/Oxen-AI/oxen-release

We have been working on a data version control tool called Oxen that is tackling many of your needs. Feel free to check it out here:

https://github.com/Oxen-AI/oxen-release#-oxen

Going down your list of requirements, Oxen has:

* Data versioning, similar paradigm to git, but built from the ground up for large ML datasets

We've been working on some open source tooling called "oxen" that was built for large datasets of images, video, audio, text etc. We wanted to solve the exact problem you're flagging here with git.

Feel free to check it out here https://github.com/Oxen-AI/oxen-release#-oxen would love any feedback!

I've used DVC in the past and generally liked its approach. That said, I wholeheartedly agree that it's clunky. It does a lot of things implicitly, which can make it hard to reason about. It was also extremely slow for medium-sized dataset (low 10s of GBs).

In response, I created a command-line tool that addresses these issues[0]. To reduce the comparison to an analogy: Dud : DVC :: Flask : Django.

[0]: https://github.com/kevin-hanselman/dud

There is also https://github.com/kevin-hanselman/dud

I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think.

[0]: https://github.com/kevin-hanselman/dud

[1]: https://kevin-hanselman.github.io/dud/benchmarks/

I don't know about replacing Make with Docker, but I use the two together to good effect. One of my favorite hacks is adding a 'docker-%' rule in my Makefile to run make commands in a Docker image[1]. It's a bit mind-bending, and there's a few gotchas, but it works surprisingly well for simple rules.

[1]: https://github.com/kevin-hanselman/dud/blob/e98de8fcdf7ad564...

Thanks for sharing your experience. It's non-trivial and surprising behavior like this that drove me to build a custom system[0] myself. When I started researching version control tools for large files, I remember feeling like git-annex and Git LFS were awkwardly bolted onto Git; Git simply wasn't designed for large files. Then I found DVC[1], and its approach rang true for me. However, after using DVC for a year or so, I grew tired of DVC's many puzzling behaviors (most of which are outlined in the README at [0]). In the end, I built the tool I wanted for the job -- one that is exceptionally simple and fast.

[0]: https://github.com/kevin-hanselman/dud