elfshaker stores binary objects efficiently
We've just added an applicability section, which explains a bit more what we do. We don't have any ELF specific heuristics.
In summary, for manyclangs, we compile with -ffunction-sections and -fdata-sections, and store the resulting object files. These are fairly robust to insertions and deletions, since the addresses are section relative, so the damage of any addresses changing is contained within the sections. A somewhat surprising thing is that this works well enough when building many revisions of clang/llvm -- as you go from commit to commit, many commits have bit identical object files, even though the build system often wants to rebuild them because some input has changed.
elfshaker packs use a heuristic of sorting all unique objects by size, before concatenating them and storing them with zstandard. This gives us an amortized cost-per-commit of something like 40kiB after compression with zstandard.
Repository hosting unofficial binary pack files for many commits of LLVM
Author here. elfshaker itself does not have a dependency on any architecture to our knowledge. We support the architectures we have immediate use of.
manyclangs provides binary pack files for aarch64 because that's what we have immediate use of. If elfshaker and manyclangs proves useful to people, I would love to see resource invested to make it more widely useful.
You can still run the manyclangs binaries on other architectures using qemu , with some performance cost, which may be tolerable depending on your use case.
Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.
Nix Packages collection
I experimented with something similar with a Linux distribution's package binary cache.
Using `bup` (deduplicating backup tool using git packfile format) I deduplicated 4 Chromium builds into the size of 1.
Large download/storage requirements for updates are one of NixOS's few drawbacks, and I think deduplication could solve that pretty much completely.
A fast high compression read-only file system
Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs
I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The compression in this case is not quite as impressive (50 GiB to around 300 MiB), but access times are typically in the millisecond region.
elfshaker: a low-footprint, high-performance version control system fine-tuned for binaries
2 projects | reddit.com/r/rust | 19 Nov 2021
Does file exists or not, this is the question
2 projects | dev.to | 16 May 2022
An automatically-updated nix-index
2 projects | reddit.com/r/NixOS | 15 May 2022
Rust nix develop & nix build - cargo2nix 0.11.0 released
2 projects | reddit.com/r/Nix | 14 May 2022
Issue with fetchurl curlOpts. Any help for a noob?
1 project | reddit.com/r/NixOS | 14 May 2022