Hystrix VS zstd

Oops, forgot to include the Hystrix link, https://github.com/Netflix/Hystrix

They had to figure out video compression that worked at the volume that they wanted to deliver. They had to build and maintain their own CDN to be able to have a always available and consistent viewing experience. Don’t even get me started on the resiliency tools like hystrix that they were kind enough to open source. I mean, they have their own fucking data science framework and they’re looking into using neural networks to downscale video.. Sound familiar? That’s cause that’s practically the same thing as Nvidia’s DLSS (which upscales instead of downscales).

When breaking up a monolithic app into microservices, the communication between these services becomes vital to the health and performance of the application. Technically, you could incorporate the features to manage this traffic directly into your application. This is what Twitter, Google, and Netflix did with massive internal libraries like Finagle, Stubby, and Hysterix.

Hystrix: https://github.com/Netflix/Hystrix Hollow: https://hollow.how/

Have you actually read the article and maybe also https://github.com/Netflix/Hystrix status section??!

I came upon Resilience4J when I was running my talk on the Circuit Breaker pattern. The talk included a demo, and it relied on Hystrix. One day, I wanted to update the demo to the latest Hystrix version and noticed that maintainers had deprecated it in favor of Resilience4J.

Netflix was talking alot about circuit breaks a few years ago, and had the Hystrix project. Looks like Hystrix is discontinued, so I'm not sure if there are good library solutions that are easy to adopt. Overall I don't see it getting talked about that frequently... beyond just exponential backoff inside a retry loop.

- https://github.com/Netflix/Hystrix

Of course, you may get different results with another dataset.

gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]

zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]

NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.

NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.

NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases

For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.

xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.

In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.

Get the data https://publicdistst.blob.core.windows.net/data/root.tar.zst magnet:?xt=urn:btih:84931cd80409ba6331f2fcfbe64ba64d4381aec5&dn=root.tar.zst How to extract https://github.com/facebook/zstd Linux (debian): `sudo apt install zstd` ``` tar -I 'zstd -d -T0' -xvf root.tar.zst ```

I've done that experiment with zstd before.

https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...

Not sure about brotli though.

If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349

The story says : "ZSTD 1.5.5 is released with a corruption fix found at Google"