stargz-snapshotter
zstd
stargz-snapshotter | zstd | |
---|---|---|
10 | 108 | |
1,048 | 22,445 | |
1.7% | 1.5% | |
8.4 | 9.7 | |
5 days ago | 7 days ago | |
Go | C | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
stargz-snapshotter
-
Tree-shaking, the horticulturally misguided algorithm
A lazy chunked delivery strategy like used in the k8s stargz-snapshotter[0] project could be effective here, where it only pulls chunks as needed, but it would probably require wasm platform changes.
[0] https://github.com/containerd/stargz-snapshotter
-
Show HN: depot.ai – easily embed ML / AI models in your Dockerfile
To optimize build speed, cache hits, and registry storage, we're building each image reproducibly and indexing the contents with eStargz[0]. The image is stored on Cloudflare R2, and served via a Cloudflare Worker. Everything is open source[1]!
Compared to alternatives like `git lfs clone` or downloading your model at runtime, embedding it with `COPY` produces layers that are cache-stable, with identical hash digests across rebuilds. This means they can be fully cached, even if your base image or source code changes.
And for Docker builders that enable eStargz, copying single files from the image will download only the requested files. eStargz can be enabled in a variety of image builders[2], and we’ve enabled it by default on Depot[3].
Here’s an announcement post with more details: https://depot.dev/blog/depot-ai.
We’d love to hear any feedback you may have!
[0] https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md
[1] https://github.com/depot/depot.ai
[2] https://github.com/containerd/stargz-snapshotter/blob/main/docs/integration.md#image-builders
[3] https://depot.dev
-
A Hidden Gem: Two Ways to Improve AWS Fargate Container Launch Times
Seekable OCI (SOCI) is a technology open-sourced by AWS that enables containers to launch faster by lazily loading the container image. It’s usually not possible to fetch individual files from gzipped tar files. With SOCI, AWS borrowed some of the design principles from stargz-snapshotter, but took a different approach. A SOCI index is generated separately from the container image and is stored in the registry as an OCI Artifact and linked back to the container image by OCI Reference Types. This means that the container images do not need to be converted, image digests do not change, and image signatures remain valid.
- containerd/stargz-snapshotter: Fast container image distribution plugin with lazy pulling
- EStargz: Lazy pull container images for faster cold starts
- How to optimize the security, size and build speed of Docker images
-
Speeding up LXC container pull by up to 3x
This is interesting and seems general purpose. Not merely for container images.
There’s this option for OCI containers which I don’t pretend to understand: https://github.com/containerd/stargz-snapshotter
It is used by containerd and nerdctl. You do have to build the image with it. Images work in OCI compatible registry. By fetching most used files first container can be started before loading is finished. Or so I gather.
-
Optimizing Docker image size and why it matters
stargz is a gamechanger for startup time. You might not need to care about image size at all
kubernetes and podmand support it, and docker support is likely coming. It lazy loads the filesystem on start-up, making network requests for things that are needed and therefore can often start up large images very fast.
https://github.com/containerd/stargz-snapshotter
-
FOSS News International #2: November 8-145, 2021
containerd/stargz-snapshotter: Fast container image distribution plugin with lazy pulling (github.com)
-
Introducing GKE image streaming for fast application startup and autoscaling
Yes, see https://github.com/containerd/stargz-snapshotter
zstd
-
Drink Me: (Ab)Using a LLM to Compress Text
> Doesn't take large amount of GPU resources
This is an understatement, zstd dictionary compression and decompression are blazingly fast: https://github.com/facebook/zstd/blob/dev/README.md#the-case...
My real-world use case for this was JSON files in a particular schema, and the results were fantastic.
-
SQLite VFS for ZSTD seekable format
This VFS will read a sqlite file after it has been compressed using [zstd seekable format](https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...). Built to support read-only databases for full-text search. Benchmarks are provided in README.
-
Chrome Feature: ZSTD Content-Encoding
Of course, you may get different results with another dataset.
gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.
NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.
NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases
For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.
xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
- Zstandard v1.5.6 – Chrome Edition
-
Optimizating Rabin-Karp Hashing
Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.
In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.
- Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
-
Cyberpunk 2077 dev release
Get the data https://publicdistst.blob.core.windows.net/data/root.tar.zst magnet:?xt=urn:btih:84931cd80409ba6331f2fcfbe64ba64d4381aec5&dn=root.tar.zst How to extract https://github.com/facebook/zstd Linux (debian): `sudo apt install zstd` ``` tar -I 'zstd -d -T0' -xvf root.tar.zst ```
-
Honey, I shrunk the NPM package · Jamie Magee
I've done that experiment with zstd before.
https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
Not sure about brotli though.
-
How in the world should we unpack archive.org zst files on Windows?
If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349
- Release Zstandard v1.5.5 · facebook/zstd
What are some alternatives?
kube-fledged - A kubernetes operator for creating and managing a cache of container images directly on the cluster worker nodes, so application pods start almost instantly
LZ4 - Extremely Fast Compression algorithm
acr - Azure Container Registry samples, troubleshooting tips and references
Snappy - A fast compressor/decompressor
containerd - An open and reliable container runtime
LZMA - (Unofficial) Git mirror of LZMA SDK releases
soci-snapshotter - A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
snoop - Snoop — инструмент разведки на основе открытых данных (OSINT world)
ZLib - A massively spiffy yet delicately unobtrusive compression library.
uChmViewer - A fork of Kchmviewer, the best software for viewing .chm (MS HTML help) and .epub eBooks.
brotli - Brotli compression format