vdo VS zstd

It is true that I have not had time to concentrate fully on this issue in the last days, and it is difficult for me to understand some of the subtleties of the subject (english is not my main language). I would like to thank you for your help and patience! I really appreciate it. FYI, I will continue to work on this issue and will update this topic. I've also asked the VDO development team to take a look at it and to tell me if they have any ideas, you can see that discussion here.

No problem, maybe someone else will read it and be able to clarify all of this. I also asked the question on VDO Github repository.

Still working on this project, I can now compile and run LVM VDO under Debian, which is a great thing (I shared the script here).

#!/bin/bash install_dependencies() { (( ${#} > 1 )) && { apt update apt install -y "$@" } || echo "No dependencies to install" } clean_dependencies() { (( ${#} > 1 )) && { apt autoremove --purge -y "$@" } || echo "No dependencies to remove" } clone_repo() { (( ${#} == 1 )) && { TEMPDIR=$(mktemp -d "${1/*\/}.XXX") git clone "$1" "${TEMPDIR}" && \ cd "$TEMPDIR" && pwd -P } } # First part : KVDO module install_dependencies git make "linux-headers-$(uname -r)" > /dev/null 2>&1 WORK_DIR="$(clone_repo https://github.com/tigerblue77/kvdo.git)" ( cd "${WORK_DIR}" && \ make -j"$(nproc)" -C "/usr/src/linux-headers-$(uname -r)" M="${WORK_DIR}" ) > kvdo_log 2>&1 cp "${WORK_DIR}/vdo/kvdo.ko" "${WORK_DIR}/uds/uds.ko" "/lib/modules/$(uname -r)" clean_dependencies git make "linux-headers-$(uname -r)" > /dev/null 2>&1 rm -Rf "${WORK_DIR}" # Second Part: VDO Module install_dependencies git make gcc uuid-dev libz-dev libdevmapper-dev libblkid-dev > /dev/null 2>&1 WORK_DIR="$(clone_repo https://github.com/dm-vdo/vdo.git)" ( cd "${WORK_DIR}" && \ make -j"$(nproc)" && \ make -j"$(nproc)" install ) > vdo_build_log 2>&1 clean_dependencies git make gcc uuid-dev libz-dev libdevmapper-dev libblkid-dev > /dev/null 2>&1 rm -Rf "${WORK_DIR}" # Last Part depmod modprobe kvdo modprobe uds

Update : the project is progressing and I managed to install VDO on Debian 11(.3) : https://github.com/dm-vdo/vdo/issues/47

lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting.

https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...

So why would anyone compress an image again by using gzip or zstd?

This is because Zstd's long-distance matcher looks for matching sequences of 64 bytes [0]. Because long matching sequences of the data will likely have the newlines inserted in different offsets in the run, this totally breaks Zstd's ability to find the long-distance match.

Ultimately, Zstd is a byte-oriented compressor that doesn't understand the semantics of the data it compresses. Improvements are certainly possible if you can recognize and separate that framing to recover a contiguous view of the underlying data.

[0] https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zs...

(I am one of the maintainers of Zstd.)

Perhaps it's worth it adding this as a note at the top of the post, maybe mentioning alternatives, such as an Actually Portable™ build of `pigz`[1] or just a windows build of zstd[2].

[1] https://cosmo.zip/pub/cosmos/tiny/pigz

[2] https://github.com/facebook/zstd/releases/latest/

If zstd give you an error and you don't handle it, the next calls may cause UB, so it kinda does both things.

https://github.com/facebook/zstd/blob/b16d193512d3ded82fd584...

( https://github.com/facebook/zstd/releases/tag/v1.5.6 )

In my opinion, it is better to check the original repository: https://github.com/inikep/lzbench

This. Even their less known work is pretty solid[1] ( used it the other day and was frankly kinda amazed at how well it performed under the circumstances ). Facebook/Meta sucks like most social madia does, but, not unlike Elon Musk, they are on the record of having some contributions to society as a whole.

[1]https://github.com/facebook/zstd

I don't think so? It's only seekable with an additional index [1], just like any other compression scheme.

[1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...