bytestring VS text-icu

The documentation is itself fairly funny, for those who don’t care to click ahead:

> This "function" has a superficial similarity to ‘unsafePerformIO’ but it is in fact a malevolent agent of chaos. It unpicks the seams of reality (and the IO monad) so that the normal rules no longer apply. It lulls you into thinking it is reasonable, but when you are not looking it stabs you in the back and aliases all of your mutable buffers. The carcass of many a seasoned Haskell programmer lie strewn at its feet.

> Witness the trail of destruction:

    https://github.com/haskell/bytestring/commit/71c4b438c675aa360c79d79acc9a491e7bbc26e7

If you bring in efficient strings from bytestring, densely packed arrays from vector, and an in-place sort from vector-algorithms, you can bring it down to 275ms (uses 19MB of mem).

bytestring

Note that this release is broken for Windows.

I've opened https://github.com/haskell/bytestring/issues/455 so the situation can be improved. You're very welcome to chime in on the discussion or to contribute some of the missing documentation yourself! :)

Highlights from the changelog:

Well, ByteString in particular should not have an IsString instance in a new report. That's pretty clear by https://github.com/haskell/bytestring/issues/140 : the concensus is that there is no good solution right now, but it should not have gotten an IsString instance in the first place. If a theoretical new Haskell Report 202x includes OverloadedStrings (as it should) to handle string literals analogously to numeric literals, I'd expect it to not give ByteString (which is really just a collection of octets) an IsString instance, with all it's issues and rattail due to the encoding question being implicitized.

Standard streaming libraries. They are being written by people that make the effort to understand performance and I have a hope that they make sure their streams run in linear space under any optimizations. It is curious and unsettling that we have standard lazy text and byte streams at the same time — and the default lazy lists, of course. I have been doing some work on byte streams and what I found out is that there is no way to check that your folds are actually space constant even if the value in question is a primitive, like say a byte — thunks may explode and then collapse over the run time of a single computation, defying any effort at inspection.

Until now, the only way to do proper Unicode sorting in Haskell was to depend on text-icu, which wraps the C library icu4c. However, there are disadvantages to depending on an external C library. In addition, the last release of text-icu was in 2015, and since then there have been changes to icu4c that cause build-failures, as noted in this issue.

text-icu occasionally breaks horribly and nondeterministically (1, 2) despite being praised by SOTU