outrun
gnu-parallel
outrun | gnu-parallel | |
---|---|---|
13 | 22 | |
3,109 | 25 | |
- | - | |
0.0 | 10.0 | |
over 1 year ago | about 9 years ago | |
Python | Perl | |
Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
outrun
-
Distcc: A fast, free distributed C/C++ compiler
While it's purpose is different it can be used to do distributed compiling, so I'll leave it here.
https://github.com/Overv/outrun
Since I was just going down this rabbit hole recently, I kind of wonder if it's possible to set the filesystem on something more like the BitTorrent protocol so things like the libraries/compilers/headers that are used during compilation dont all need to come from the main pc. It probably wouldn't be useful until you reached a stupid number of computers and you started reaching the limits of the Ethernet wire, but for something stupid that can run on a pi cluster it would be a fun project.
-
Programing laptop
Your mention of compile heavy workloads reminded me of a project called Outrun, it offloads work to another machine. All it seems to require is Python, Fuse3 and ssh.
-
The u-root CPU command
Awesome! This write up is satisfyingly detailed. Prior work in this space includes Plan9 of course, as well as the python project Outrun, which has it's own RPC-based FUSE FS: https://github.com/Overv/outrun
Other approachs to deployment in particular include the functional package managers Nix and Guix, which can create lightweight application images, and could probably be cobbled together into some sort of remote environment replication even across architectures. As I read on, I thought less about how this compares with Guix in regards to application/environment packaging and more about how these things could be glued together in interesting ways, because I think the intro leads in through slightly off-label examples, if that makes sense. Application packaging isn't what this addresses at the end of the day, but it's no less fascinating for it.
- GitHub - Overv/outrun: Execute a local command using the processing power of another Linux machine.
- Way to run commands using other linux system's compute power
- Outrun - Execute a local command using the processing power of another Linux machine.
gnu-parallel
-
SQL query execution idea
You can use GNU Parallel (https://www.gnu.org/software/parallel/) to run command-line clients with all of those queries. You can set up the upper limit of simultaneous clients run, and this will automatically handle all possible parallelism.
- Parallel – shell tool for executing jobs in parallel using one or more computers
-
Distcc: A fast, free distributed C/C++ compiler
Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines with multiple cores.
1) set up passwordless, ssh.
and
2) use the gnu parallel. https://www.gnu.org/software/parallel/
gnu parallel is super flexible, very useful.
-
Peplum: F/OSS distributed parallel computing and supercomputing at Home with Ruby infrastructure
How does this stack up againg GNU parallel? If you just wanna parallelize CLI work-loads (like nmap), parallel should be easier, I guess.
-
Search in your Jupyter notebooks from the CLI, fast.
It requires jq for JSON processing and GNU parallel for concurrent searches in the notebooks.
- Is there a way to use all CPU cores while using RIBlast?
-
Can cuda help me here?
Since you've got lots of images, you could use GNU Parallel to spread the job across multiple CPUs.
-
5 great Perl scripts to keep in your sysadmin toolbox
Gnu parallel
- Is there an .deb package for installing GNU parallel?
-
Modern SPAs without bundlers, CDNs, or Node.js
You could easily use something like GNU Parallel:
https://www.gnu.org/software/parallel/
What are some alternatives?
rffmpeg - rffmpeg: remote SSH FFmpeg wrapper tool
Parallel
OpenAFS - Fork of OpenAFS from git.openafs.org for visualization
bazel-buildfarm - Bazel remote caching and execution service
icecream - Distributed compiler with a central scheduler to share build load
lolcate-rs - Lolcate -- A comically fast way of indexing and querying your filesystem. Replaces locate / mlocate / updatedb. Written in Rust.
llama
xidel - Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
cargo-mutants - :zombie: Inject bugs and see if your tests catch them!
jc - CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
remote-apis - An API for caching and execution of actions on a remote system.
ripgrep - ripgrep recursively searches directories for a regex pattern while respecting your gitignore