libu8ident VS featurebase

In programming languages it's much worse. Identifiers can either be unidentifiable, and if so everybody has a different opinion what "identifiable" means. Even the standard on identifiers, UTF-39, is buggy and has too many interpretations, leading to a complete disaster. https://github.com/rurban/libu8ident/blob/master/doc/c11.md

In punycode domain names it's quite simple still.

With other names, it's even worse. No-one cares. Linkers do not, username and filesystem drivers do not. The Apple HFS+ did care a bit one day, until someone in the higher ranks decided that no-one needs unicode security anymore and switched the new APFS to unsafe again.

No, it's definitely not safe to use unrestricted Unicode in a compiler. See https://github.com/rurban/libu8ident/ for identifier rules, and http://www.unicode.org/reports/tr55/ for much worse problems.

That's why unicode published the security guidelines and mechanisms to avoid such attacks. In 2004 already.

The problem is that nobody cared. Browsers invented punycode instead of following tr39, email ditto. But ok, at least something. Java did it, cperl did, rust did it.

Everybody else is vulnerable. Esp. most other programming languages, filesystems and login systems. https://github.com/rurban/libu8ident/blob/master/doc/c11.md

I'm maintaining such a library.

coreutils, diff, grep, patch, sed and friends all cannot find Unicode strings, they have no string support. They can only mimic filesystems, finding binary garbage. Strings are so rthi g different than pure ASCII or BINARY garbage. Strings have an encoding and are Unicode.

Filesystems are even worse because they need to treat filenames as identifiers, but do not. Nobody cares about TR31, TR39, TR36 and so on.

Here is an overview of the sad state of Unicode unsafeties in programming languages: https://github.com/rurban/libu8ident/blob/master/c11.md

You can try https://www.featurebase.com/, they have an open source version but they are focusing on the cloud offering

Featurebase[1] is a database based on bitmaps. (Formerly known as Piloasa) I'm not sure if they actually use Roaring Bitmaps [2] or they implemented it themselves. Either way it's based on the idea of compressed bitmaps.

[1] https://github.com/FeatureBaseDB/featurebase

If this still sounds complex, or if you want to see what happens when some smart people make it even faster, then give it a try with FeatureBase.