rules_python VS josh

What's SV?

I honestly don't know why anyone would use that... as in what does Bazel do better than virtually anything else that can provide this functionality. But, I used to be an ops engineer in a big company which wanted everything to be Maven, regardless of whether it does it well or not. So we built and deployed with Maven a lot of weird and unrelated stuff.

Not impossible, but not anything I'd advise anyone to do on their free time.

Specifically wrt' the link you posted, if you look here: https://github.com/bazelbuild/rules_python/blob/main/python/... it says that only pure Python wheels are supported, but that's also a lie, they don't support half of the functionality of pure Python wheels.

So, definitely not worth using, since lots of functionality is simply not there.

Regarding bazel, the rules_python has a py_wheel rule that helps you creating wheels that you can upload to pypi (https://github.com/bazelbuild/rules_python/blob/52e14b78307a...).

If you want to see an approach of bazel to pypi taken a bit to the extreme you can have a look at tensorflow on GitHub to see how they do it. They don't use the above-mentioned building rule because I think their build step is quite complicated (C/C++ stuff, Vida/ROCm support, python bindings, and multiOS support all in one before you can publish to pypi).

Python support in Bazel now looks more promising with `rules_python`: https://github.com/bazelbuild/rules_python

`rules_go` to my understanding is great too.

Over years, Bazel is not as opinionated as before, mostly because adoptions in different orgs force it to be so.

I have personally run converted build systems to Bazel, and use it for personal projects as well.

Bazel 1.0 was released in October 2019. If you were using it "a few years ago", I'm guessing you were using a pre-1.0 version. There's not some cutoff where Bazel magically got easy to use, and I still wouldn't describe it as "easy", but the problem it solves is hard to solve well, and the community support for Bazel has gotten a lot better over the past years.

https://github.com/bazelbuild/rules_python

The difficulty and complexity of using Bazel is highly variable. I've seen some projects where using Bazel is just super simple and easy, and some projects where using Bazel required a massive effort (custom toolchains and the like).

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") #------------------------------------------------------------------------------ # Python #------------------------------------------------------------------------------ # enable python rules http_archive( name = "rules_python", url = "https://github.com/bazelbuild/rules_python/releases/download/0.2.0/rules_python-0.2.0.tar.gz", sha256 = "778197e26c5fbeb07ac2a2c5ae405b30f6cb7ad1f5510ea6fdac03bded96cc6f", )

Why use submodules when you can properly vendor the upstream git, and export/import commits without breaking hashes on either side?

https://github.com/josh-project/josh

We've been using josh at TVL for years and it's just amazing.

Scalar explicitly does not implement the virtualized filesystem the OP is referring to. The original Git VFS for Windows that Microsoft designed did in fact do this, but as your second link notes, Microsoft abandoned that in favor of Scalar's totally different design which explicitly was about scaling repositories without filesystem virtualization.

There's a bunch of related features they added to Git to achieve scalability without virtualization. Those are all useful and Scalar is a welcome addition. But the need for a virtual filesystem layer for large-scale repositories is still a very real one. There are also some limitations that aren't ideal; for example Git's partial clones IIRC can only be used as a "cone" applied to the original filesystem hierarchy. More generalized designs would allow mapping any arbitrary paths in the original repository to any other path in the virtual checkout. Tools like Josh can do this today with existing Git repositories[1]. That helps you get even sparser and smaller checkouts.

The Git for Windows that was referenced isn't even that big at 300GB, by the way. Game studios regularly have repositories that exist at multi-terabyte size, and they have also converged on similar virtualization solutions. For example, Destiny 2 uses a "virtual file synchronization" layer called VirtualSync[2] that reduced the working size of their checkouts by over 98%, multiple terabytes of savings per person. And in a twist of fate, VirtualSync was implemented thanks to a feature called "ProjFS" that Microsoft added to Windows... which was motivated originally by the Git VFS for Windows they abandoned!

[1] https://github.com/josh-project/josh

[2] https://www.gdcvault.com/play/1027699/Virtual-Sync-Terabytes...

Good overview, I know these sorts of pains well. Lots of hard questions and few definitive wins/right answers. How to organize a massive repository out in the open is still an open question. On that note, recently, I've been experimenting with this project called josh, which basically is like 'git subtree on extreme steroids, functioning as a git proxy':

https://josh-project.github.io/josh/

It basically lets you unify/view many repositories as a single one, or equivalent to split a mono-repo into smaller sized units of work for CI, specific teams, etc. It's bidirectional, so you push and pull from josh and everything goes into a single linear history in the mono repo. And because it's bidirectional, people in the mono-repo can still do things like make large-scale atomic changes across all sub-repositories, and those get reflected.

Josh currently isn't suitable for a lot of workloads due to various reasons (authentication is one that stands out), but it's actually the first tool I have seen that manages to offer BitKeeper-like "subtrees" that work really well, at scale, for large repos and teams. It requires some care to make sure "sub-trees" can be usable units of work, but it was one of the best features of BK in my opinion and really great for people doing one-off contributions, or isolating trees/changes to specific developers.

I'd be interested to know if there are other open alternatives to this. It's a nice point in the design space between solutions like "integrate with the filesystem layer to do sparse clones" or "just split up the repos."

With regard to repo composition, I have been following this project: https://github.com/josh-project/josh