Confusables
libu8ident
Confusables | libu8ident | |
---|---|---|
5 | 9 | |
146 | 17 | |
- | - | |
0.0 | 1.8 | |
over 1 year ago | 10 months ago | |
Python | C | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Confusables
-
The Revenge of Unicode
I wrote a small python library for searching obfuscated Unicode text by creating regex expressions; our company’s chat was being overrun by spammers.
https://github.com/wanderingstan/Confusables
From the readme:
E.g. "℮1೦" would match "Hello"
"Hello" gets turned into the following regex of character classes:
[HHℋℌℍΗⲎНᎻᕼꓧ𐋏ⱧҢĦӉӇ]
-
Unicode Utilities: Confusables
You're right. The code was correct but the Readme hadn't been updated to reflect the lack of backslashes. Thanks for the find!
https://github.com/wanderingstan/Confusables/commit/c4121428...
- Unicode normalizing library to parse attacker text as English
-
You either die an MVP or live long enough to build content moderation
I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO.
https://github.com/wanderingstan/Confusables
E.g. "𝓗℮𝐥1೦" would match "Hello"
libu8ident
- Roaring bitmaps are compressed bitmaps, can be 100x faster
-
International domain names: where does HTTPS://meßagefactory.ca lead you?
In programming languages it's much worse. Identifiers can either be unidentifiable, and if so everybody has a different opinion what "identifiable" means. Even the standard on identifiers, UTF-39, is buggy and has too many interpretations, leading to a complete disaster. https://github.com/rurban/libu8ident/blob/master/doc/c11.md
In punycode domain names it's quite simple still.
With other names, it's even worse. No-one cares. Linkers do not, username and filesystem drivers do not. The Apple HFS+ did care a bit one day, until someone in the higher ranks decided that no-one needs unicode security anymore and switched the new APFS to unsafe again.
-
Using Unicode in a compiler
No, it's definitely not safe to use unrestricted Unicode in a compiler. See https://github.com/rurban/libu8ident/ for identifier rules, and http://www.unicode.org/reports/tr55/ for much worse problems.
- Ask HN: What interesting problems are you working on? ( 2022 Edition)
- Unicode Utilities: Confusables
-
How can you be fooled by the U+202E trick?
That's why unicode published the security guidelines and mechanisms to avoid such attacks. In 2004 already.
The problem is that nobody cared. Browsers invented punycode instead of following tr39, email ditto. But ok, at least something. Java did it, cperl did, rust did it.
Everybody else is vulnerable. Esp. most other programming languages, filesystems and login systems. https://github.com/rurban/libu8ident/blob/master/doc/c11.md
- Prevent Trojan Source attacks with GCC 12
-
Unicode Normalization Forms: When ö = ö
I'm maintaining such a library.
coreutils, diff, grep, patch, sed and friends all cannot find Unicode strings, they have no string support. They can only mimic filesystems, finding binary garbage. Strings are so rthi g different than pure ASCII or BINARY garbage. Strings have an encoding and are Unicode.
Filesystems are even worse because they need to treat filenames as identifiers, but do not. Nobody cares about TR31, TR39, TR36 and so on.
Here is an overview of the sad state of Unicode unsafeties in programming languages: https://github.com/rurban/libu8ident/blob/master/c11.md
- Why does Windows 10 run faster than Fedora?
What are some alternatives?
stream.new - The repo for https://stream.new
featurebase - A crazy fast analytical database, built on bitmaps. Perfect for ML applications. Learn more at: http://docs.featurebase.com/. Start a Docker instance: https://hub.docker.com/r/featurebasedb/featurebase
redditfs - An interactive command line utility to save files and directories to Reddit.
libredwg - Official mirror of libredwg. With CI hooks and nightly releases. PR's ok
confusables - A nodejs library for removing confusable unicode characters from strings.
safeclib - safec libc extension with all C11 Annex K functions
nbperf - Improved NetBSD's Perfect Hash Generation Tool v3
reals - A lightweight python3 library for arithmetic with real numbers.
ts-pg-orm - Delightful Typescript PostgreSQL ORM
m4b-tool - m4b-tool is a command line utility to merge, split and chapterize audiobook files such as mp3, ogg, flac, m4a or m4b
poly - A Go package for engineering organisms.
RoaringBitmap - A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others