base32768 VS Twitter Text Obj

qntm did a fun project of using larger bases, constrained to subsets of unicode instead of ASCII like base64. It's specifically for social channels where you're constrained by the number of code points, but not bytes, so you want to maximize data per code point. base2048 is pretty impressive, and base32768 is just absurd.

base32768 is ideal for storing binary data in localStorage with an efficiency of 94%.

A similar thing is actually practical in JavaScript which mandates that all strings are UTF-16. You can cram more data into strings in memory if you use base-327168 encoding, and it serializes to equally compact JSON.

I recall that Twitter allowed 140 Chinese characters in tweets originally, but when they switched to 280 ascii characters, CJK languages were not included. The current documentation does say that some languages and emojis use 2 characters per symbol of the 280 limit, limiting those languages to 140. https://developer.twitter.com/en/docs/counting-characters

I assume you we’re going for half of the (new) character limit? That wouldn’t be bad as a rough guess, except Twitter uses UTF-8 encoding and what counts as a character is a bit complicated. Certain objects like usernames in a reply or image URLs hosted on Twitter are not counted, either. The maximum size of a modern tweet could therefore be upwards of 280 x 4 = 1,120 bytes.

Given the examples in this article it seems like it could potentially be used for this!

It turns out, working this out is non-trivial, as Twitter has a few rules around how it count's characters. These are basically:

Japanese, Korean and Chinese were excluded from that increased cap, because they already had a significant advantage in being able to put more information into a single character. (More specifically, it's implemented such that ideograms count as 2 characters now, and emoji do, too.)

The 280 character limit in a tweet isn't equal to the number of glyphs in a tweet.

https://developer.twitter.com/en/docs/counting-characters