zfs VS lz4_flex

> The only non-junk PCIe3 option that's even advertised here recently is the overpriced WD Red SN700.

Those WD drives seem to have some real issues, at least with ZFS and btrfs. :(

https://github.com/openzfs/zfs/discussions/14793

IMO the "next fs" is just zfs. They somewhat recently merged RAIDZ expansion feature https://github.com/openzfs/zfs/pull/12225 and make regular improvements. If no file system has what you need today, zfs will probably be the first one to have it "tomorrow," imo.

https://github.com/openzfs/zfs/issues/15526#issuecomment-181...

> zpool get all tank | grep bclone

> kc3000 bcloneused 442M

> kc3000 bclonesaved 1.42G

> kc3000 bcloneratio 4.30x

> My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.

A couple years ago, I had an idea for convincing a filesystem to go faster using 2 compression steps instead of one. I couldn't see why it wouldn't work, and I also couldn't convince myself it should.

It seems to have worked out. [1]

[1] - https://github.com/openzfs/zfs/commit/f375b23c026aec00cc9527...

https://github.com/openzfs/zfs/issues/7631

This is a long-standing issue with zvols which affects overall system stability, and has no real solution as of yet.

TIL. Thank you! https://github.com/openzfs/zfs/pull/15022

Difference may be significant, yes https://github.com/openzfs/zfs/pull/14243

I run a benchmark based on my LZ4 implementation (de/compressor) between cax11 and cx11 since the instances cost the same. The code doesn't use any SIMD.

By the way, the PR removing bounds checks from extend_from_within_overlapping is up: https://github.com/PSeitz/lz4_flex/pull/141

Zap uses LZ4 as it's primary compression algorithm, although I might add ZSTD as an option too. The LZ4 crate i'm using goes over the compression better than I can in the span of a comment. Check it out here.

I wrote something here how hc could work: https://github.com/PSeitz/lz4_flex/issues/21

Yes, the updated benchmarks are here: https://github.com/pseitz/lz4_flex#results-v080-17-05-2021-safe-decode-and-safe-encode-off

I ported the block format to Rust matching the C implementation in performance and ratio.

https://github.com/pseitz/lz4_flex

Following this change count_same_bytes is unsound - it offsets the pointer by the value of candidate without any bounds checks, which may result in out-of-bounds access.