zfec
zpaqlpy
zfec | zpaqlpy | |
---|---|---|
2 | 2 | |
373 | 19 | |
0.3% | - | |
7.7 | 10.0 | |
6 months ago | over 1 year ago | |
Python | Rust | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zfec
-
The Haskell job market has been growing steaily since 2008
We used reflex-frp, so our app was a webview that worked on localhost and Android. The docs say it also works on iOS but we don't have an iPhone.
The process was learning Functional Reactive Programming, then learning reflex-frp, then getting a contract with obsidian (creators of reflex) for one hour a week where we could ask questions.
( https://github.com/reflex-frp/reflex-platform )
We had a grant requirement to create a phone client for Tahoe-LAFS, a Python application with a bunch of dependencies, including ZFEC, a forward error correction library.
( https://tahoe-lafs.readthedocs.io/ )
( https://github.com/tahoe-lafs/zfec/ )
We needed bug for bug compatibility with the Python codebase, so I ran Tahoe on localhost and tested the Haskell client against the Python server. We used servant to build the API, since it builds both client and server side from the same description.
( https://hackage.haskell.org/package/servant )
-
Xz format considered inadequate for long-term archiving
I disagree with the premise of the article. Archive formats are all inadequate for long-term resilience and making them adequate would be a violation of the “do one thing and do it right” principle.
To support resilience, you don’t need an alternative to xz, you need hashes and forward error correction. Specifically, compress your file using xz for high compression ratio, optionally encrypt it, then take a SHA-256 hash to be used for detecting errors, then generate parity files using PAR[1] or zfec[2] to correct errors.
[1] https://wiki.archlinux.org/title/Parchive
[2] https://github.com/tahoe-lafs/zfec
zpaqlpy
-
Xz format considered inadequate for long-term archiving
ZPAQ is the name of the tool but ZPAQ is also the name of the container format that gets used. ZPAQ embeds the decompression algorithm in the archive. One could store zstd-compressed blocks in ZPAQ archives as soon as a zpaql decompressor exists (e.g., for brotli there is a slow one implemented in a python subset and compiled to zpaql https://github.com/pothos/zpaqlpy).
I don't know exactly whether other formats are better for seeking and streaming, but since the baseline is tar, ZPAQ (in the 2.0 spec) is already better as it supports deduplication and files can even be updated append-only, and the compression is not an afterthought wrapped around it but well integrated.
- QOI – The Quite OK Image Format
What are some alternatives?
pixz - Parallel, indexed xz compressor
zstd - Zstandard - Fast real-time compression algorithm
reflex-platform - A curated package set and set of tools that let you build Haskell packages so they can run on a variety of platforms. reflex-platform is built on top of the nix package manager.
BorgBackup - Deduplicating archiver with compression and authenticated encryption.
fossa-action - The action sets up and caches the latest release of fossa-cli, infer the correct configuration from the current system state, analyze the project for a list of its dependencies, and upload the results to FOSSA.
restic - Fast, secure, efficient backup program
haskell-jobs-statistics
fpng - Super fast C++ .PNG writer/reader
fpnge - Demo of a fast PNG encoder.
dwarfs - A fast high compression read-only file system for Linux, Windows and macOS
libjxl - JPEG XL image format reference implementation
qoi - The “Quite OK Image Format” for fast, lossless image compression