simdutf
frozen
simdutf | frozen | |
---|---|---|
11 | 10 | |
960 | 1,210 | |
4.8% | - | |
9.1 | 6.1 | |
3 days ago | about 1 month ago | |
C++ | C++ | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
simdutf
- Glibc Buffer Overflow in Iconv
-
Vectorizing Unicode conversions on real RISC-V hardware
The project was mostly inspired by simdutf [0] which has been around for a couple of years already, and I don't think iconv has any of its vectorized implementations for other architectures.
[0] https://github.com/simdutf/simdutf
-
Cray-1 performance vs. modern CPUs
I'm actually doing something quite similar in my, in progress, unicode conversion routines.
For utf8 validation there is a clever algorithm that uses three 4-bit look-ups to detect utf8 errors: https://github.com/simdutf/simdutf/blob/master/src/icelake/i...
Aside on LMUL, if you haven't encountered it yet: rvv allows you to group vector registers when configuring the vector configuration with vsetvl such that vector instruction operate on multiple vector registers at once. That is, with LMUL=1 you have v0,v1...v31. With LMUL=2 you effectively have v0,v2,...v30, where each vector register is twice as large. with LMUL=4 v0,v4,...v28, with LMUL=8 v0,v8,...v24.
In my code, I happen to read the data with LMUL=2. The trivial implementation would just call vrgather.vv with LMUL=2, but since we only need a lookup table with 128 bits, LMUL=1 would be enough to store the lookup table (V requires a minimum VLEN of 128 bits).
So instead I do six LMUL=1 vrgather.vv's instead of three LMUL=2 vrgather.vv's because there is no lane crossing required and this will run faster in hardware: (see [0] for a relevant mico benchmark)
# codegen for equivalent of that function
-
What C++ library do you wish existed but hasn’t been created yet?
utf8 normalization, stemming, case insensitive comparison. https://github.com/unicode-rs example for rust What are options for C++? 1. translate to utf16 ( https://github.com/simdutf/simdutf ) and use icu -- slow 2. boost text, https://github.com/tzlaine/text , also slow (because the author doesn't care or couldn't care), we made a lot of patches to make our library faster than lucene, but still this part is slower than icu for utf16 (icu for utf16 also very slow...)
-
[Preprint] Transcoding Unicode Characters with AVX-512 Instructions
You can find the corresponding assembly code in this repository. The main branch only contains implementations based on C++ with intrinsics.
-
What's everyone working on this week (10/2023)?
The next big thing is making it LSP-compatible. All language servers must implement UTF-16 based character offsets, which is kinda unfortunate considering that files are much more likely to be stored in UTF-8 (I think?). I don't want to do the UTF-8 -> UTF-16 transcoding, so instead I'll use the excellent simdutf library to count how much code points a UTF-8 string would take if it was transcoded into UTF-16 — which is much faster than actual transcoding. So this is what I'm going to do this week — rewriting parsers to produce UTF-16 offsets + some final benchmarking. After that is done, I'll consider the "research" part of this project completed and will start writing an actual Markdown parser.
-
Why would a language not natively support SIMD?
You can find the assembly code here: https://github.com/simdutf/simdutf/tree/clausecker The corresponding C++ code is in the main branch.
- High speed Unicode routines using SIMD
-
text-2.0-rc1 with UTF8 underlying representation is available for testing!
Or via an ultrafast simdutf.
- Simdutf: Unicode validation and transcoding at billions of characters per second
frozen
-
Making a "constant mapping"
I found this extension that implements "frozen" versions of some C++ containers, but I was wondering if there is a good solution available in the standard library.
-
Static map - is it possible?
A library exists that can produce constexpr hash table based containers.
-
What C++ library do you wish existed but hasn’t been created yet?
I use the Frozen library for that. Since the conversions should be known at compile time you can make constexpr hash tables for lookups.
-
Command-line util for class implementation (My first try at a professional c++ application)
The constexpr dependency of note here is frozen.
-
Ambition is cute.
In C++, a drop-in replacement for your DSA can provide significant improvements over the standard library. Particularly the standard unordered_map class can be improved by 50% to 100% (e.g. https://github.com/greg7mdp/parallel-hashmap, or for static maps https://github.com/serge-sans-paille/frozen). Of course, recognize that creating a DS/A from scratch is an entire project, and you shouldn't roll your own for an independent codebase.
-
[Hobby] Bomberman fan 2D Animator needed
Technologies (for curious folks): C++17, SFML, Entt, Frozen, Protobuf, spdlog, GoogleTest, GoogleBenchmark, CMake and Dear ImGui for debug purpose.
-
May 2021 monthly "What are you working on?" thread
In the language, I added anonymous array literals. I did some cleanup in the compiler and updated to LLVM 12 from 10 (which was pretty trivial, surprisingly). I also added frozen, a C++ perfect-hashing library, as a dependency to speed up the lookup of keywords in my lexer. The library exploits C++’s constexpr features to generate a perfect hash at compile-time without any separate build step, which is great, and it also provides a drop-in replacement for std::unordered_map that uses the hash.
-
MSVC Backend Updates in Visual Studio 2019 version 16.10 Preview 2 | C++ Team Blog
This is where I plug Frozen :-] https://github.com/serge-sans-paille/frozen
-
What (relatively) easily to implement features would you like to see in c++23.
I’ve no idea how hard it is to implement, but return type polymorphism would be nice. Especially returning different things based on the constexpress of the result. And then add Frozen eqivalents of associative containers to the STL, so that, for example constexpr auto set = std::make_set(...) would be frozen::set, and auto set = std::make_set(...) would be std::set.
-
Compile-time INI config parsing and accessing with C++20
In which case, I believe the answer your question would be yes: the frozen map.
What are some alternatives?
simdutf8 - SIMD-accelerated UTF-8 validation for Rust.
gram_grep - Search text using a grammar, lexer, or straight regex. Chain searches for greater refinement.
DirectXMath - DirectXMath is an all inline SIMD C++ linear algebra library for use in games and graphics apps
parallel-hashmap - A family of header-only, very fast and memory-friendly hashmap and btree containers.
simde - Implementations of SIMD instruction sets for systems which don't natively support them.
STL - MSVC's implementation of the C++ Standard Library.
eve - Expressive Vector Engine - SIMD in C++ Goes Brrrr
bluebird - A work-in-progess programming language modeled after Ada and C++
Vc - SIMD Vector Classes for C++
mpv - 🎥 Command line video player
simdjson - Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks
c3c - Compiler for the C3 language