encoding_rs
simdutf8
Our great sponsors
| encoding_rs | simdutf8 | |
|---|---|---|
| 8 | 15 | |
| 357 | 509 | |
| - | 1.2% | |
| 6.3 | 1.2 | |
| 4 days ago | 4 days ago | |
| Rust | Rust | |
| GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
encoding_rs
-
This Program is Illegally Packaged in 14 Distributions
Author of ripgrep here.
Maybe, umm, don't depend on auto-detection tools as the ultimeate source of whether something is appropriately licensed or not? encoding_rs is clearly licensed, otherwise I wouldn't have used it in ripgrep: https://github.com/hsivonen/encoding_rs
Take it up with the auto-detection tools. Or at least do your due diligence before claiming ripgrep is using "unlicensed" code.
- Learning to Parse Text Files: BOM!
-
Yore - library for decoding/encoding character sets according to OEM code pages
You might also want to include tests, https://github.com/hsivonen/encoding_rs/tree/master/src/test_data could be used as a reference.
-
Show HN: High-speed UTF-8 validation in Rust
That's not the only use of SIMD in the crate (e.g. see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...), but I haven't looked into exactly where/how it's used further.
simdutf8
-
simdutf: Unicode validation and transcoding at billions of characters per second
That's not enough to make it interesting. There's already a porting of it in rust (https://github.com/rusticstuff/simdutf8), and inclusion in the stdlib has already been discussed: the problem is that you can't use simd in all supported targets and conditional compilation/detection is also very tricky.
-
Making Rust Float Parsing Fast: libcore Edition
No, libcore uses simple branching code at the moment, see https://github.com/rust-lang/rust/issues/68455. The issue is still actively being worked on. Note, it's not a simple drop in, and there seem to be even faster algorithms. For now there is https://github.com/rusticstuff/simdutf8.
-
What's everyone working on this week (19/2021)?
I will work on simdutf8, either
- simdutf8 v0.1.2 - Apple Silicon can get very fast UTF-8 validation too
-
simdutf v0.1.1 - A small step for semver, one giant leap for performance.
Now I have to benchmark again, it might be negligable. See also the discussion for this pull request.
-
Show HN: High-speed UTF-8 validation in Rust
Check the benchmarks section (https://github.com/rusticstuff/simdutf8#Benchmarks), second table. simdutf8 is up to 28 % faster on my Comet Lake CPU. However with pure ASCII clang does something magical with simdjson and it beats my implementation by a lot. GCC-compiled simdjson is slower all around except for a few outliers with short byte sequences.
The algorithm is the one from simdjson, the main difference is that it uses an extra step in the beginning to align reads to the SIMD block size.
- High-speed UTF-8 validation in Rust
What are some alternatives?
serde - Serialization framework for Rust
sqloxide - Python bindings for sqlparser-rs
simdjson - Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks
simdutf - Unicode routines (UTF8, UTF16, UTF32) and Base64: billions of characters per second using SSE2, AVX2, NEON, AVX-512, RISC-V Vector Extension. Part of Node.js and Bun.
icu4x - Solving i18n for client-side and resource-constrained environments.
cxx - Safe interop between Rust and C++
bumpalo - A fast bump allocation arena for Rust
plover - Open source stenotype engine
fast_float - Fast and exact implementation of the C++ from_chars functions for number types: 4x to 10x faster than strtod, part of GCC 12 and WebKit/Safari
fontdue - The fastest font renderer in the world, written in pure rust.
feel