gnu-parallel
icecream
gnu-parallel | icecream | |
---|---|---|
22 | 16 | |
25 | 1,553 | |
- | 0.6% | |
10.0 | 0.0 | |
about 9 years ago | 5 months ago | |
Perl | C++ | |
GNU General Public License v3.0 only | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
gnu-parallel
-
SQL query execution idea
You can use GNU Parallel (https://www.gnu.org/software/parallel/) to run command-line clients with all of those queries. You can set up the upper limit of simultaneous clients run, and this will automatically handle all possible parallelism.
- Parallel – shell tool for executing jobs in parallel using one or more computers
-
Distcc: A fast, free distributed C/C++ compiler
Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines with multiple cores.
1) set up passwordless, ssh.
and
2) use the gnu parallel. https://www.gnu.org/software/parallel/
gnu parallel is super flexible, very useful.
-
Peplum: F/OSS distributed parallel computing and supercomputing at Home with Ruby infrastructure
How does this stack up againg GNU parallel? If you just wanna parallelize CLI work-loads (like nmap), parallel should be easier, I guess.
-
Search in your Jupyter notebooks from the CLI, fast.
It requires jq for JSON processing and GNU parallel for concurrent searches in the notebooks.
- Is there a way to use all CPU cores while using RIBlast?
-
Can cuda help me here?
Since you've got lots of images, you could use GNU Parallel to spread the job across multiple CPUs.
-
5 great Perl scripts to keep in your sysadmin toolbox
Gnu parallel
- Is there an .deb package for installing GNU parallel?
-
Modern SPAs without bundlers, CDNs, or Node.js
You could easily use something like GNU Parallel:
https://www.gnu.org/software/parallel/
icecream
- Icecream: Distributed compiler with a central scheduler to share build load
-
Distcc: A fast, free distributed C/C++ compiler
Related
https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case.
https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.
-
Do you use ccache to speed up compilation times
Of course! The github readme provides a lot of info - https://github.com/icecc/icecream
- GitHub - icecc/icecream: Distributed compiler with a central scheduler to share build load
-
Ccache – a fast C/C++ compiler cache
If you like distcc, did you ever give icecc a try?
https://github.com/icecc/icecream
I never had the time to set it up properly, but by the looks of it, it should be even better.
- People who use distributed builds, how do you handle many compilers?
- Fuchsia Workstation
-
Give local devices a way to connect to clients? - openvpn
I would like to have a icecc setup I can vpn into. It seems that with normal configs the clients can talk to the scheduler, but the scheduler cant connect to the clients as it tries to connect to the device running the openvpn server not the one behind it. How could I make my openvpn clients appear almost as physical devices on the network, with unique IP's that local devices can connect to; or if that is unnecessary how could I solve this?
-
ccache 4.6 released
Glad to see a new release on this! I've read worrying news about the state of icecc, and the followup uncertain news on sccache, so I hope at least some part of the tooling is in a good shape.
-
Best way to manage dependencies with c++?
I always wanted to try to use cmake-conan so I could let Conan grab all packages but have a neat cmake script being in charge of what gets built when. Also, this would allow me to easily switch between CMake fetchcontent and Conan packages that may or may not be stashed automatically on a local Artifactory server. Secondly, since now all build requirements are stashed on a server and binary reproducible, you could concider adding icecream and ccache into the mix. (Try running a node one of your buildservers for massive speadups with icecream) This however does require a reproducible build environment (by configure script) which conan again is really good in.
What are some alternatives?
Parallel
sccache - Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
bazel-buildfarm - Bazel remote caching and execution service
ccache - ccache – a fast compiler cache
lolcate-rs - Lolcate -- A comically fast way of indexing and querying your filesystem. Replaces locate / mlocate / updatedb. Written in Rust.
keppel - Regionally federated multi-tenant container image registry
xidel - Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
compiler-benchmark - Benchmarks compilation speeds of different combinations of languages and compilers.
jc - CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
gg - The Stanford Builder
ripgrep - ripgrep recursively searches directories for a regex pattern while respecting your gitignore
cmake-init-conan-example - cmake-init generated executable project with Conan integration