Halide VS smhasher

If CPU/GPU execution speed is the goal while simultaneously code golfing the source size, https://halide-lang.org/ might have come in handy.

This is a task where Halide https://halide-lang.org/ could really shine! It disconnects logic from scheduling (unrolling, vectorizing, tiling, caching intermediates etc), so every step the author describes in the article is a tunable in halide. halide doesn't appear to have bindings for golang so calling C++ from go might be the only viable option.

Probably would have been much easier to do 15 times in https://halide-lang.org/

The idea behind Halide is that scheduling memory access patterns is critical to performance. But, access patterns being interwoven into arithmetic algorithms makes them difficult to modify separately.

So, in Halide you specify the arithmetic and the schedule separately so you can rapidly iterate on either.

It is not the sorting per-se which was improved here, but sorting (particularly short sequences) on modern CPUs with really the complexity being on the difficulty of predicting what will work quickly on these modern CPUs.

Doing an empirical algorithm search to find which algorithms fit well on modern CPUs/memory systems is pretty common, see e.g. FFTW, ATLAS, https://halide-lang.org/

Halide https://halide-lang.org/

It doesn’t apply in this case, but in general if you really want the best vectorization I would suggest using https://halide-lang.org instead of trying to coerce your compiler.

If we drop the "APL" requirement, wouldn't Halide fit your criteria for the third?

The algorithm passes all SMHasher quality tests and uses rounds of AES block cipher internally, so it is quite robust! For comparison XxH3, t1ha0 and many others don't pass SMHasher (while being slower).

Confirmed, I tested it. https://github.com/rurban/smhasher

There's lots of great hash functions out there: some are super fast, like xxhash and highly optimized, others are also super fast umash and based on interesting math ideas from finite fields^1, while maintaining high quality (according to SMHasher). Others are also fast and interesting (tabulation hash, that may sometimes be seemingly universal), one of the main originators of those ideas are Mikkel Thorup^2. Anyway, a couple of years ago I also tried my hand at building hashes and created a few that passed SMHasher (tifuhash ~ a floating point hash, beamsplitter - a seemingly-universal tabulation style hash, and this one discohash - a "more traditional" ARX-based design (addition rotation xor)^3 ).

0: https://github.com/rurban/smhasher/blob/master/xxh3.h

1: https://pvk.ca/Blog/2022/12/29/fixing-hashing-modulo-alpha-e...

2: https://arxiv.org/abs/1505.01523

3: https://eprint.iacr.org/2018/898.pdf https://crypto.polito.it/content/download/480/2850/file/docu...

4: https://en.wikipedia.org/wiki/BLAKE_(hash_function)

Discohash (posted here) is the fastest one I made, it's simple and doesn't rely on any arch-specific optimizations or vector instructions (AVX etc ~ tho I suppose...they could be added? I'm definitely no expert in them, I barely get away with doing the C/C++ implementations!)

The main mixing round function is:

  mix(const int A) {

ubsan, asan, valgrind tests are missing. some do offer symbolic verification of the algo, but not the implementations.

See my https://github.com/rurban/smhasher#crypto paragraph, and

The spinach story reminds me a lot on the false recommendation of siphash for hash table DDOS prevention. https://github.com/rurban/smhasher#security

The authors came up in their widely cited paper with a proper solution to spread the random hash seed into the inner loop, vastly enhancing its security by avoiding trivial hash collision attacks. But a secure, slow hash function can never prevent from normal hash seed attacks, when the random seed is known somehow. esp. with dynamic languages it's trivial to get the seed externally.

Other trivial countermeasures must be used then, which also don't make hash tables 10x slower, keeping them practical.

This is the best, most comprehensive hash test suite I know of: https://github.com/rurban/smhasher/

you might want to particularly look into murmur, spooky, and metrohash. I'm not exactly sure of what the tradeoffs involved are, or what your need is, but that site should serve as a good starting point at least.

Here is a good comparison table, as you can see, BLAKE can perform in secure way much faster than crc32, so my original point, - to use non weak hashes unless you really have a reason/requirement not to do so

smhasher is a great place to testing results for a massive number of hash algorithms.