Zstd

Open-source projects categorized as Zstd

Top 23 Zstd Open-Source Projects

  • 7-Zip-zstd

    7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard

    Project mention: WinRAR musste shady werden. | /r/de | 2023-12-10
  • compress

    Optimized Go Compression Packages

    Project mention: Show HN: Gogosseract, a Go Lib for CGo-Free Tesseract OCR via Wazero | news.ycombinator.com | 2023-11-04

    There's a pure-go zstd at https://github.com/klauspost/compress - it's likely faster than running the upstream zstd under Wazero.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • libarchive

    Multi-format archive and compression library

    Project mention: WinRAR musste shady werden. | /r/de | 2023-12-10
  • pgBackRest

    Reliable PostgreSQL Backup & Restore

    Project mention: pgBackRest: PostgreSQL S3 backups | dev.to | 2023-08-10

    This tutorial explains how to backup PostgreSQL database using pgBackRest and S3.

  • dwarfs

    A fast high compression read-only file system for Linux, Windows and macOS

    Project mention: Help! Does anyone know how to install johncena141 games on linux? | /r/LinuxCrackSupport | 2023-07-01

    on a fresh install all you need is dwarfs https://github.com/mhx/dwarfs and libopenal1

  • lzbench

    lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

    Project mention: My experience with btrfs so far | /r/linuxmasterrace | 2023-05-07

    Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench

  • p7zip

    A new p7zip fork with additional codecs and improvements (forked from https://sourceforge.net/projects/sevenzip/ AND https://sourceforge.net/projects/p7zip/).

    Project mention: Ubuntu 23.04 (Lunar Lobster) | /r/linux | 2023-04-20

    nearly every main distro I am aware of has both available. The reason you still see p7zip is because the CLI incompatibilities vs the newer 7z/7zip executables and the general licensing issues. Most users of "old p7zip" are actually using the actively maintained https://github.com/p7zip-project/p7zip which is updated, supporting unix permissions and zstd and so on.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • squashfs-tools

    tools to create and extract Squashfs filesystems

  • lizard

    Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster. (by inikep)

  • ratarmount

    Access large archives as a filesystem efficiently, e.g., TAR, RAR, ZIP, GZ, BZ2, XZ, ZSTD archives

    Project mention: Show HN: Rapidgzip – Parallel Gzip Decompressing with 10 GB/S | news.ycombinator.com | 2023-09-04
  • python-zstandard

    Python bindings to the Zstandard (zstd) compression library

  • TurboBench

    Compression Benchmark

    Project mention: Zstd Content-Encoding planned to ship with Chrome 123 | news.ycombinator.com | 2024-02-07

    I'm still unconvinced about this addition. And I don't even dislike Zstandard.

    The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2].

    According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead. In fact, Meta employs the main author of Zstandard so it would have been easier to tune Zstandard instead of Brotli. While Brotli has some fundamental difference from Zstandard (in particular Brotli doesn't use arithmetic-equivalent coding), no one has concretely demonstrated that difference would prevent Brotli from being fast enough for dynamic contents in my opinion.

    [1] https://issues.chromium.org/issues/40196713

    [2] https://github.com/powturbo/TurboBench/issues/43

    [3] https://docs.google.com/document/d/14dbzMpsYPfkefAJos124uPrl...

  • hpcc-js-wasm

    HPCC-Systems Web-Assembly (JavaScript)

  • zstd-rs

    zstd-decoder in pure rust

  • squashfs-tools-ng

    A new set of tools and libraries for working with SquashFS images

    Project mention: C Strings and my slow descent to madness | news.ycombinator.com | 2023-04-06

    ... except that that is also subtly broken.

    It works if you write multiple UTF-8 code-units in one go, but breaks if you send them in several writes, or if you use the ANSI API (with the A suffix). Guess what the Windows implementation of stdio (printf and friends) does.

    I already had some fun with this: https://github.com/AgentD/squashfs-tools-ng/issues/96#issuec...

    And we didn't even discuss command line argument passing yet :-)

    I tried to test it with the only other two languages I know besides English: German and Mandarin. Specifically also, because the later requires multi-byte characters to work. Getting this to work at all in a Windows terminal on an existing, German Windows 7 installation was an adventure on it's own.

    Turns out, trying to write language agnostic command line applications on Windows is a PITA.

  • zramd

    Automatically setup swap on zram ✨ with optional systemd support, a simpler alternative to zram-generator and systemd-swap

  • wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

  • zstd-seekable-format-go

    Seekable ZSTD compression format implemented in Golang.

  • zstd.cr

    Crystal bindings to the Zstandard (zstd) compression library

  • zreader

    Read compressed NDJSON .zst files easily

    Project mention: 📣 I want to debunk Reddit's claims, and talk about their unwillingness to work with developers, moderators, and the larger community, as well as say thank you for all the support | /r/apolloapp | 2023-06-19
  • ZRA

    ZStandard Random Access (ZRA) allows random access inside an archive compressed using ZStandard

  • destream

    A tool & Python 3 library to decompress anything

  • xtarfile

    Wrapper around tarfile to add support for more compression formats

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-07.

Zstd related posts

Index

What are some of the best open-source Zstd projects? This list will help you:

Project Stars
1 7-Zip-zstd 4,572
2 compress 4,453
3 libarchive 2,780
4 pgBackRest 2,145
5 dwarfs 1,244
6 lzbench 827
7 p7zip 727
8 squashfs-tools 688
9 lizard 633
10 ratarmount 565
11 python-zstandard 454
12 TurboBench 307
13 hpcc-js-wasm 292
14 zstd-rs 224
15 squashfs-tools-ng 184
16 zramd 83
17 wget-lua 79
18 zstd-seekable-format-go 68
19 zstd.cr 39
20 zreader 30
21 ZRA 28
22 destream 12
23 xtarfile 7
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com