zfec VS zpaqlpy

We used reflex-frp, so our app was a webview that worked on localhost and Android. The docs say it also works on iOS but we don't have an iPhone.

The process was learning Functional Reactive Programming, then learning reflex-frp, then getting a contract with obsidian (creators of reflex) for one hour a week where we could ask questions.

( https://github.com/reflex-frp/reflex-platform )

We had a grant requirement to create a phone client for Tahoe-LAFS, a Python application with a bunch of dependencies, including ZFEC, a forward error correction library.

( https://tahoe-lafs.readthedocs.io/ )

( https://github.com/tahoe-lafs/zfec/ )

We needed bug for bug compatibility with the Python codebase, so I ran Tahoe on localhost and tested the Haskell client against the Python server. We used servant to build the API, since it builds both client and server side from the same description.

( https://hackage.haskell.org/package/servant )

I disagree with the premise of the article. Archive formats are all inadequate for long-term resilience and making them adequate would be a violation of the “do one thing and do it right” principle.

To support resilience, you don’t need an alternative to xz, you need hashes and forward error correction. Specifically, compress your file using xz for high compression ratio, optionally encrypt it, then take a SHA-256 hash to be used for detecting errors, then generate parity files using PAR[1] or zfec[2] to correct errors.

[1] https://wiki.archlinux.org/title/Parchive

[2] https://github.com/tahoe-lafs/zfec

ZPAQ is the name of the tool but ZPAQ is also the name of the container format that gets used. ZPAQ embeds the decompression algorithm in the archive. One could store zstd-compressed blocks in ZPAQ archives as soon as a zpaql decompressor exists (e.g., for brotli there is a slow one implemented in a python subset and compiled to zpaql https://github.com/pothos/zpaqlpy).

I don't know exactly whether other formats are better for seeking and streaming, but since the baseline is tar, ZPAQ (in the 2.0 spec) is already better as it supports deduplication and files can even be updated append-only, and the compression is not an afterthought wrapped around it but well integrated.