xxHash VS meow_hash

> GPU Hash Table?

How bad would performance have suffered if you sha256'd the lines to build the map? I'm going to guess "badly"?

Maybe something like this in CUDA: https://github.com/Cyan4973/xxHash ?

Examples of fast, simple hashes that are independent enough includes murmur, xxHash, Fowler–Noll–Vo hash function and many others

If you're only using the hash for non-cryptographic applications, there are much faster hashes: https://github.com/Cyan4973/xxHash

I found the benchmark result of AMD ryzen 5950X

But what's the catch, you're thinking ? Well, it is a bit slower than its counterparts when it comes to deserializing (and marginally faster for serializing). To achieve smaller footprint, it uses a few tricks and notably a custom hash table to deduplicate strings. This comes at a cost of course (even when featuring xxHash to speed things up), but keeps the slowdown reasonable (I think).

Non cryptographic hashes has collisions, for example, assume you having content like "abcdefg" which hashed value is "123", in case of weak hash algorithm some other content like "abcdefZ" can also have a hash "123" which basically means such hash function is failed to be unique fingerprint of particular content. BLAKE3 for example can do 6-7Gb/s which make it pretty fast and secure. If your requirement accepts collision with defined error rate, I would advise you to take a look at XXH3 if you need very snappy hash algorithm, which can run at pace or RAM access (30GB/s+), but again, run tests at particular equipment you targeting, may be AES hardware accelerated MeowHash will serve you better.

I'm late on the reply but I was using xxhash for something similar, but found that Meow hash was faster for me. Feel free to benchmark, since I'm sure it could vary depending on CPU architecture.

for everyone who doesn't think highly of Casey Muratori (or at least the way he conducts himself online), the author of Meow Hash, took the criticism quite graciously:

https://twitter.com/cmuratori/status/1417546500083568641

https://github.com/cmuratori/meow_hash/issues/80

Is a standardized hashing algorithm required? SHA256 takes several CPU cycle per byte, but there are several hashing algorithm utilizing AES-NI that gives them 15-16 bytes per cycles, which translates to tens of gigabytes per second on modern CPU. At least one of them has been ported to .NET Core, and from what I see properly calls AES primitives.

MeowHash is the fastest non-cryptographic hash available online, and if it were used in CPython, which performs a hash for every fundamental operation in the language, then Python would be much faster.

Meow hash claims 3-4x faster hashing over this, still passes smhasher, and is a few years old. https://mollyrocket.com/meowhash

Have you seen the meow hash implementation? Would that be suitable for this ?