repairwheel VS jbig2dec

I got into cross-compiling Python wheels (e.g., building macos wheels on linux and vice versa). Zig's `zig cc` does much of the heavy lifting, but one step in building a portable wheel is the "repair" process which vends native library dependencies into the wheel, necessitating binary patching (auditwheel does this for linux, delocate for macos).

I wanted to be able to do this cross platform, so I re-implemented ELF patching and Mach-O patching and adhoc signing in Python, and wrapped them into a tool called repairwheel: https://github.com/jvolkman/repairwheel

I'll plug some work I've been doing to (attempt to) enable cross compilation of Python wheels. I put together a small example [1] that builds the zstandard wheel, and can build macos wheels on linux and linux wheels on macos using zig cc.

macos wheels must still be adhoc signed (codesign) and binary patched (install_name_tool), so I re-implemented those functions in Python [2].

[1] https://github.com/jvolkman/bazel-pycross-zstandard-example

[2] https://github.com/jvolkman/repairwheel/tree/main/src/repair...

I mixed auditwheel, delocate, and delvewheel into a single tool called repairwheel and reimplemented all of the required external tools (patchelf, otool, codesign, etc.) in pure python.

> The worst offender (so far) is the JBIG2 format (several major libraries, including jbig2dec), a very popular format that gets EXTREMELY high compression ratios on bilevel images of types typical to scanned pdfs. But: it's also a format that's pretty slow to decompress—not something you want in a UI loop, like a PDF reader is! And, there's no way around that—if you look at the hot loop, which is arithmetic coding, it's a mess of highly branchy code that's purely serial and cannot be thread- nor SIMD- parallelized.

Looking at the jbig2dec code, there appears to be some room for improvement. If my observations are correct, each segment has its own arithmetic decoder state, and thus can be decoded in its own thread. The main reader loop[1] is basically a state machine which attempts to load each segment in sequence[2], but it should not need to. The file has segment headers which contains the segments offsets and sizes. It should be possible to first read this header, then spawn N-threads to decode N-segment in parallel. Obviously, you don't want the threads competing for the file resource, so you could load each segment into its own buffer first, or mmap the whole file into memory.

[1]:https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

[2]:https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

I still haven't found an tolerably fast PDF reader, and I'm permanently miserable with that file format. The example in OP works great, but that's only an "easy, modern" PDF made up of text. There's still nothing adequate (mupdf/mutool included) for the common case of scanned-page PDF's.

The root problem isn't an easy performance fix: it's that a very popular PDF image compression format, JBIG2 [0], is unlike modern formats slow in decompression as well as compression. Here's the decompress hot loop [1,2] from libjbig2dec.so, which MuPDF calls out to. I isn't thread- or SIMD- parallelized, and I suspect that it isn't possible at all. There's just no easy way forwards in the near future—other than "buy faster CPU's".

[0] https://en.wikipedia.org/wiki/JBIG2

[1] https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

[2] https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...