Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
sccache
Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
Related
https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case.
https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.
Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later.
I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants, a mutation testing tool for Rust.
It can be. By default it is local.
But it has protobufs interfaces (IIRC), so a distributed build farm would generate the grpc endpoints for their implementation and then you tell bazel on the command line (or via .bazelrc) the address of the build farm it can use.
There's a couple of projects that implement the distributed/grpc part, the main one is https://github.com/bazelbuild/bazel-buildfarm
Related
https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case.
https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.
https://github.com/mozilla/sccache is another option which addresses the use cases of both icecream and ccache (and also supports Rust, and cloud storage of artifacts, if those are useful for you)
Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your cache accurate.
On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.
Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.
Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.
While it's purpose is different it can be used to do distributed compiling, so I'll leave it here.
https://github.com/Overv/outrun
Since I was just going down this rabbit hole recently, I kind of wonder if it's possible to set the filesystem on something more like the BitTorrent protocol so things like the libraries/compilers/headers that are used during compilation dont all need to come from the main pc. It probably wouldn't be useful until you reached a stupid number of computers and you started reaching the limits of the Ethernet wire, but for something stupid that can run on a pi cluster it would be a fun project.
Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines with multiple cores.
1) set up passwordless, ssh.
and
2) use the gnu parallel. https://www.gnu.org/software/parallel/
gnu parallel is super flexible, very useful.