gnu-parallel VS bazel-buildfarm

You can use GNU Parallel (https://www.gnu.org/software/parallel/) to run command-line clients with all of those queries. You can set up the upper limit of simultaneous clients run, and this will automatically handle all possible parallelism.

Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines with multiple cores.

1) set up passwordless, ssh.

and

2) use the gnu parallel. https://www.gnu.org/software/parallel/

gnu parallel is super flexible, very useful.

How does this stack up againg GNU parallel? If you just wanna parallelize CLI work-loads (like nmap), parallel should be easier, I guess.

It requires jq for JSON processing and GNU parallel for concurrent searches in the notebooks.

Since you've got lots of images, you could use GNU Parallel to spread the job across multiple CPUs.

Gnu parallel

You could easily use something like GNU Parallel:

https://www.gnu.org/software/parallel/

It can be. By default it is local.

But it has protobufs interfaces (IIRC), so a distributed build farm would generate the grpc endpoints for their implementation and then you tell bazel on the command line (or via .bazelrc) the address of the build farm it can use.

There's a couple of projects that implement the distributed/grpc part, the main one is https://github.com/bazelbuild/bazel-buildfarm

Not sure what to say concerning buildfarm's remote execution.

Reading an issue I've opened 1y ago [1], seems the backend requires the client to have a specific gcc version.

That's a strong limitation imho.

[1] https://github.com/bazelbuild/bazel-buildfarm/issues/545

You can also set up a remote build service that uses the same environment you run the containers in using something like https://github.com/bazelbuild/bazel-buildfarm

Currently migrating to Bazel + bazel remote execution via Bazel buildfarm. Bazel has native support for build clusters.