opentype-shaping-documents VS Duktape

> The tangent in (1) on how they contradict unicode could have been skipped as well

Not only because confusables already exist, but also (as I already said[1] the previous time this was posted) covering all ligatures used in all typographical styles is very much a non-goal of Unicode. The official position is that the font shaping layer[2] sits atop Unicode’s semantic representation and is free to ligate, spindle, or mutilate it for display however it prefers (at least for Latin, Greek, and Cyrillic it’s a preference; other scripts can’t be rendered at all without doing it, such as Arabic—barring the legacy presentational forms—or Burmese[3]).

The only reason Unicode even has those ligatures is that some IBM encodings (which were more presentational in nature) encoded them, and IBM employees wrote a large part of the early standard (based on the decades of i18n experience they had at that point) and wanted roundtripping.

[1] https://news.ycombinator.com/item?id=29639966

[2] https://github.com/n8willis/opentype-shaping-documents

[3] https://r12a.github.io/scripts/mymr/my.html#combiningV

> Off the top of my head, I don't know of a terminal that actually implements the entire (very complex) set of Unicode text rendering behaviors

There are likely two problems with this:

First, nobody actually seems to know how bidirectional text should interact with terminal control sequences, or indeed how it should be typeset on a terminal in the first place (where are the paragraph boundaries?). There is the pre-Unicode bi-directional support mode (BDSM, I kid you not) in ECMA-48[1] and TR/53[2], which AFAIK nobody implements nor cares about, and which doesn’t seem to actually; there are terminal emulators made by bidi-language users[3], which AFAIK nobody has written down the behaviour of; there is the Freedesktop bidi terminal spec[4], which is a draft and AFAIK nobody implements yet either but at least some people care about; finally, there are bidi-language users who say that spec is a mistake[5].

Second, aside from bidi and a smattering of other things such as emoji, there is no detailed “Unicode rendering behaviour”, there are only standards specific to font formats, the most recent being OpenType, which is dubiously compatible across implementations, decently documented only through painstaking reverse engineering (sometimes in words[6], sometimes only in Freetype library code), and generally full of snakes[7]. And it has no notion of monospace font—only of a (proportional) font where all Lat/Cyr/Grk characters just happen to have the same advance.

AFAICT that is not an oversight or negligence, but rather a concession to the fact that there are scripts which don’t really have a notion of monospace in the typographic tradition and in fact are written such that it’s extremely unclear what monospace would even mean—certainly not one or two cells per codepoint (e.g. Burmese or Tibetan; apparently there are Arabic monospace fonts[8] but I’ve no idea how the hell they work). Not coincidentally, those are the scripts where you need that shaper, otherwise nothing works.

[1] https://www.ecma-international.org/publications-and-standard...

[2] https://www.ecma-international.org/publications-and-standard...

[3] https://news.ycombinator.com/item?id=8086417

[4] https://terminal-wg.pages.freedesktop.org/bidi/

[5] http://litcave.rudi.ir/

[6] https://github.com/n8willis/opentype-shaping-documents

[7] https://litherum.blogspot.com/2019/03/addition-font.html

[8] https://news.ycombinator.com/item?id=10395464

Im my projects I search for single file libs.(like https://github.com/svaarala/duktape etc...)

Use lrand48(), or better, implement a high-quality RNG like PCG or splitmix64.

You can also refer to the Unicode routines of other small JS engines[1,2], those don’t use ICU either, although the implementations are mercilessly size-optimized (to put it politely) and restricted to what the target JS version requires (e.g. casemapping but no normalization).

[1] https://github.com/bellard/quickjs/blob/master/libunicode.c

[2] https://github.com/svaarala/duktape/blob/master/src-input/du...

> memcpy from bytes to a NaN should work fine

Signaling NaNs are explicitly undefined in C11 F.2.1.: "This specification does not define the behavior of signaling NaNs." - and in practice may be "quieted" by conversion to Quiet NaNs, changing their bit patterns. Fast math optimization flags will also break the hell out of your code by assuming NaNs are impossible. I want to say there are more circumstances where optimizers and compiler generated code can butcher your NaN payloads, but I'd be working off recollected hearsay and I can't find a source, so don't quote me on that.

NaN boxing is common enough that, if you take the right precautions, a modern compiler should probably support it, maybe. NaN boxing is uncommon enough that, if your codebase needs to be sufficiently portable, you need an opt out for when it breaks. Let's review duktape's scars:

https://github.com/svaarala/duktape/blob/123d9426d5e5b36d5da...

https://github.com/svaarala/duktape/blob/5252b7a50611a3cb8bf...

https://github.com/svaarala/duktape/blob/224a0b89ca08a36e37e...

Note that "the right precautions" involve unions and proper integer types to avoid optimizer-invoked rewrites of the value and debugging when things go wrong, not simply YOLOing bytes into a double via memcpy. Note that debugging when it all goes terribly wrong can be quite painful. I've personally had the misfortune of being forced to debug duktape being built with fast math optimizatoins enabled on one "rare" platform + build configuration that wasn't caught by duktape's #if defined(__FAST_MATH__) checks linked above (wasn't Clang nor GCC, so go figure it didn't make the same #define)

I was expecting this to be about Duktape <https://github.com/svaarala/duktape>, but heh, for sure no. I'd bet $1 there's no way youtube-dl would switch, but I wonder if yt-dlp would?

Fast math optimizations can break code like this by breaking isNaN.

I was porting a C++ project to a certain platform - and that platform enabled a -ffast-math equivalent by default in Release (but not Debug) builds! This broke duktape, a JS engine said project embedded, in some nasty and subtle ways. Instead of storing a number/pointer/??? (8 bytes) + type tag (4? bytes) for each dynamically typed JS value, duktape can bit-pack values into a single 8 byte "double" value by storing object/string handles as NaN values - this isn't an uncommon trick for dynamically typed scripting stuff:

https://github.com/svaarala/duktape/blob/c3722054ea4a4e50f48...

Naturally, the -ffast-math equivalent broke isNaN checks, which caused random object/string handles to be mistakenly reinterpreted as "numbers" - but only in Release builds, for this one particular platform, in one rarely taken branch, so neither QA nor CI caught it, leading to hours of manufacturing a repro case, stepping through an absurd amount of code, and then finally looking at the default build rules and facepalming.

Cursing the platform vendor under my breath, I overrode the defaults to align with the defaults of every other config x platform combination we already had: no fast math. If you want those optimizations, use SSE-friendly NaN-avoiding intrinsics - or, if you must use the compiler flags, ensure you do so consistently across build configs and platforms, perhaps limited to a few TUs or modules if possible. This allows you to have a chance at using your Debug builds to debug the resulting "optimizations".

Sure. For example, DukTape is an implementation of Javascript designed to be embedded in other projects. Google's V8 Javascript engine (used in Chrome), can also be embedded, see Node.Js for example.

- Duktape (4.8k stars)