babelfish_extensions VS jbig2dec

[2] https://github.com/babelfish-for-postgresql/babelfish_extens...

DBeaver works surprisingly nicely with less popular DBs. I work with Babelfish for PostgreSQL [1], it supports connections with SQL Server client libs. Most GUI client tools (like SSMS) expect "real" SQL Server on the other end of the wire - depend on various system views for DB introspection, so only partially work with Babelfish. Even if client tool is based on JDBC (like SQuirell SQL), it doesn't guarantee that this tool won't use additional SQL Server-specific querues for i trospection. DBeaver is much better at this, I guess it is using JDBC API or DB-neutral INFORMATION_SCHEMA views.

[1] https://babelfishpg.org/

Was just about to write that AWS does that, but no, it's only for MS SQL Server: https://babelfishpg.org/, https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide....

Migration to Postgres from MS SQL Server doesn't have to be a complete app rewrite. Obviously any .Net components would be a problem, but if it's just a matter of existing clients and T-SQL, there's always Babelfish.

https://babelfishpg.org/

> moving some data from SQL Server to Postgres

I don't have any first-hand experience with Postgres replication to share, just, when moving DB from MSSQL, Babelfish extensions for Postgres (https://babelfishpg.org/) may be of interest.

> The worst offender (so far) is the JBIG2 format (several major libraries, including jbig2dec), a very popular format that gets EXTREMELY high compression ratios on bilevel images of types typical to scanned pdfs. But: it's also a format that's pretty slow to decompress—not something you want in a UI loop, like a PDF reader is! And, there's no way around that—if you look at the hot loop, which is arithmetic coding, it's a mess of highly branchy code that's purely serial and cannot be thread- nor SIMD- parallelized.

Looking at the jbig2dec code, there appears to be some room for improvement. If my observations are correct, each segment has its own arithmetic decoder state, and thus can be decoded in its own thread. The main reader loop[1] is basically a state machine which attempts to load each segment in sequence[2], but it should not need to. The file has segment headers which contains the segments offsets and sizes. It should be possible to first read this header, then spawn N-threads to decode N-segment in parallel. Obviously, you don't want the threads competing for the file resource, so you could load each segment into its own buffer first, or mmap the whole file into memory.

[1]:https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

[2]:https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

I still haven't found an tolerably fast PDF reader, and I'm permanently miserable with that file format. The example in OP works great, but that's only an "easy, modern" PDF made up of text. There's still nothing adequate (mupdf/mutool included) for the common case of scanned-page PDF's.

The root problem isn't an easy performance fix: it's that a very popular PDF image compression format, JBIG2 [0], is unlike modern formats slow in decompression as well as compression. Here's the decompress hot loop [1,2] from libjbig2dec.so, which MuPDF calls out to. I isn't thread- or SIMD- parallelized, and I suspect that it isn't possible at all. There's just no easy way forwards in the near future—other than "buy faster CPU's".

[0] https://en.wikipedia.org/wiki/JBIG2

[1] https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...

[2] https://github.com/ArtifexSoftware/jbig2dec/blob/master/jbig...