janet-utf8 VS text

Regarding UTF-8 encoding:

“And a couple of important consequences:

- You CAN’T determine the length of the string by counting bytes.

- You CAN’T randomly jump into the middle of the string and start reading.

- You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.”

One of the things I had to get used to when learning the programming language Janet is that strings are just plain byte sequences, unaware of any encoding. So when I call `length` on a string of one character that is represented by 2 bytes in UTF-8 (e.g. `ä`), the function returns 2 instead of 1. Similar issues occur when trying to take a substring, as mentioned by the author.

As much as I love the approach Janet took here (it feels clean and simple and works well with their built-in PEGs), it is a bit annoying to work with outside of the ASCII range. Fortunately, there are libraries that can deal with this issue (e.g. https://github.com/andrewchambers/janet-utf8), but I wish they would support conversion to/from UTF-8 out of the box, since I generally like Janet very much.

One interesting thing I learned from the article is that the first byte can always be determined from its prefix. I always wondered how you would recognize/separate a unicode character in a Janet string since it may have 1-4 bytes length, but I guess this is the answer.

It sounds like a generic length function in Unicode in 2023 is no longer a good idea. These articles complaining about the variety of lengths in Unicode are annoying at this point. Pretty much all of them can be summed up as, "Well, it depends." And, that isn't wrong. But nerds love to argue until they are blue in the face about the One Correct Answer. Sheesh.

This is the most interesting comparison article I have seen in years about Unicode processing in C++: https://thephd.dev/the-c-c++-rust-string-text-encoding-api-l...

The author is also the lead on an open source C++ Unicode library called ztd.txt: https://github.com/soasis/text

How does this compare with https://github.com/tzlaine/text and https://github.com/soasis/text ?

You might consider https://github.com/soasis/text (https://ztdtext.readthedocs.io/en/latest/index.html) which is a proof of concept of this proposal that we may get in C++26

from this link for the tested option that was the only one to check all the boxes green in the article... it makes me ask myself if the article was written biased with the purpose of promoting ztd.text and that these boxes/criteria were to an extent cherry-picked with that purpose in mind.

Github: https://github.com/soasis/text

That's a huge misrepresentation of the reality. Of course it has its own string class, because there has not been a proper type in the standard library for Unicode text, and for many, we are not there yet (otherwise we would not be having other libraries filling the empty space).

You can checkout ztd.text, which implements presented ideas. Few years passed, so they were more polished too.

The library you talk about is https://github.com/soasis/text and he has an account here.

Project Link: https://github.com/soasis/text Announcement/Blog Link: https://thephd.dev/any-encoding-ever-ztd-text-unicode-cpp

PS: found some example code in the github project which seems to take care of the special Windows situation:

https://github.com/soasis/text/blob/main/examples/documentat...

Unfortunately this doesn't look so simple anymore.