simdutf8 VS stdarch

That's not enough to make it interesting. There's already a porting of it in rust (https://github.com/rusticstuff/simdutf8), and inclusion in the stdlib has already been discussed: the problem is that you can't use simd in all supported targets and conditional compilation/detection is also very tricky.

No, libcore uses simple branching code at the moment, see https://github.com/rust-lang/rust/issues/68455. The issue is still actively being worked on. Note, it's not a simple drop in, and there seem to be even faster algorithms. For now there is https://github.com/rusticstuff/simdutf8.

I will work on simdutf8, either

Now I have to benchmark again, it might be negligable. See also the discussion for this pull request.

Check the benchmarks section (https://github.com/rusticstuff/simdutf8#Benchmarks), second table. simdutf8 is up to 28 % faster on my Comet Lake CPU. However with pure ASCII clang does something magical with simdjson and it beats my implementation by a lot. GCC-compiled simdjson is slower all around except for a few outliers with short byte sequences.

The algorithm is the one from simdjson, the main difference is that it uses an extra step in the beginning to align reads to the SIMD block size.

Good to know! How would you compare it with the std_detect implementation, underlying the standard library (https://github.com/rust-lang/stdarch/tree/master/crates/std_detect)?

Fortunately, @burntsushi (thanks) is sharing the workaround in the rust-lang github issue. The trick is to reinterpret a (non-tagged) union for this.

The stdarch tests will test target-specific stuff like SIMD while to UI tests are more generic. They both test that the compiler will produce an executable that produce the correct output (or that rustc itself will produce the correct output).

I made enough progress to be able to compile the tests of stdarch and most of them pass: that's a huge milestone!

You can check the code. Apparently the std implementation uses the OSXSAVE register to check that the OS supports saving AVX/AVX2 registers during context switches and only then enables it. In a non-std context one might not generally be able to depend on the OSXSAVE register.