Twitter Text Obj VS base2048

I recall that Twitter allowed 140 Chinese characters in tweets originally, but when they switched to 280 ascii characters, CJK languages were not included. The current documentation does say that some languages and emojis use 2 characters per symbol of the 280 limit, limiting those languages to 140. https://developer.twitter.com/en/docs/counting-characters

I assume you we’re going for half of the (new) character limit? That wouldn’t be bad as a rough guess, except Twitter uses UTF-8 encoding and what counts as a character is a bit complicated. Certain objects like usernames in a reply or image URLs hosted on Twitter are not counted, either. The maximum size of a modern tweet could therefore be upwards of 280 x 4 = 1,120 bytes.

Given the examples in this article it seems like it could potentially be used for this!

It turns out, working this out is non-trivial, as Twitter has a few rules around how it count's characters. These are basically:

Japanese, Korean and Chinese were excluded from that increased cap, because they already had a significant advantage in being able to put more information into a single character. (More specifically, it's implemented such that ideograms count as 2 characters now, and emoji do, too.)

The 280 character limit in a tweet isn't equal to the number of glyphs in a tweet.

https://developer.twitter.com/en/docs/counting-characters

qntm did a fun project of using larger bases, constrained to subsets of unicode instead of ASCII like base64. It's specifically for social channels where you're constrained by the number of code points, but not bytes, so you want to maximize data per code point. base2048 is pretty impressive, and base32768 is just absurd.

Numbers don't have to loop until 32k or 2048 (or even up to 1,112,064, until such point the Unicode standard allows for more)

On a semi-related note, they mention base64 encoding messages to evade filters. There were actually other base{n} methods [1] created specifically for Twitter to be more space optimized though not as readily available to operating systems. I guess this is less useful if they are really expanding the text limit to 4k soon but figured I would add it in the event they add a parser for base64.

[1] - https://github.com/qntm/base2048

Actually, in the case of twitter they do some weird counting. It is mostly based on codepoints, true, but some codepoints are considered "heavy" and are counted twice, see https://github.com/qntm/base2048

I know of one case of Twitter doing client side validation [1]. Maybe there are more?

1 - https://github.com/qntm/base2048#note

Base 2048\ (15 comments)