bazel-buildfarm
gnu-parallel
Our great sponsors
| bazel-buildfarm | gnu-parallel | |
|---|---|---|
| 4 | 22 | |
| 624 | 25 | |
| 4.5% | - | |
| 9.5 | 10.0 | |
| 5 days ago | about 9 years ago | |
| Java | Perl | |
| Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
bazel-buildfarm
-
Distcc: A fast, free distributed C/C++ compiler
It can be. By default it is local.
But it has protobufs interfaces (IIRC), so a distributed build farm would generate the grpc endpoints for their implementation and then you tell bazel on the command line (or via .bazelrc) the address of the build farm it can use.
There's a couple of projects that implement the distributed/grpc part, the main one is https://github.com/bazelbuild/bazel-buildfarm
-
Rust Is Portable
Not sure what to say concerning buildfarm's remote execution.
Reading an issue I've opened 1y ago [1], seems the backend requires the client to have a specific gcc version.
That's a strong limitation imho.
[1] https://github.com/bazelbuild/bazel-buildfarm/issues/545
-
Running container_push inside a container
You can also set up a remote build service that uses the same environment you run the containers in using something like https://github.com/bazelbuild/bazel-buildfarm
-
Distributed build clusters
Currently migrating to Bazel + bazel remote execution via Bazel buildfarm. Bazel has native support for build clusters.
gnu-parallel
-
SQL query execution idea
You can use GNU Parallel (https://www.gnu.org/software/parallel/) to run command-line clients with all of those queries. You can set up the upper limit of simultaneous clients run, and this will automatically handle all possible parallelism.
- Parallel – shell tool for executing jobs in parallel using one or more computers
-
Distcc: A fast, free distributed C/C++ compiler
Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines with multiple cores.
1) set up passwordless, ssh.
and
2) use the gnu parallel. https://www.gnu.org/software/parallel/
gnu parallel is super flexible, very useful.
-
Peplum: F/OSS distributed parallel computing and supercomputing at Home with Ruby infrastructure
How does this stack up againg GNU parallel? If you just wanna parallelize CLI work-loads (like nmap), parallel should be easier, I guess.
-
Search in your Jupyter notebooks from the CLI, fast.
It requires jq for JSON processing and GNU parallel for concurrent searches in the notebooks.
- Is there a way to use all CPU cores while using RIBlast?
-
Can cuda help me here?
Since you've got lots of images, you could use GNU Parallel to spread the job across multiple CPUs.
-
5 great Perl scripts to keep in your sysadmin toolbox
Gnu parallel
- Is there an .deb package for installing GNU parallel?
-
Modern SPAs without bundlers, CDNs, or Node.js
You could easily use something like GNU Parallel:
https://www.gnu.org/software/parallel/
What are some alternatives?
llama
Parallel
mainframer - Tool for remote builds. Sync project to remote machine, execute command, sync back.
lolcate-rs - Lolcate -- A comically fast way of indexing and querying your filesystem. Replaces locate / mlocate / updatedb. Written in Rust.
pyroscope-rs - Pyroscope Profiler for Rust. Profile your Rust applications.
xidel - Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
sccache - Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
jc - CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
rules_closure - Closure rules for Bazel
ripgrep - ripgrep recursively searches directories for a regex pattern while respecting your gitignore
pyzmq - PyZMQ: Python bindings for zeromq
parallel - xargs for concurrent, distributed execution of shell commands