BLAKE3 VS xxHash

> might be easier with a public domain license instead of the current ones

There reference implementation is public domain (CC0) or at your choice Apache 2.0

https://github.com/BLAKE3-team/BLAKE3/blob/master/LICENSE

Fyi, blake3 was released in 2019 and should probably be used over blake2 unless you have some strong reason not to. It's basically a reimplementation of blake2 with performance tweaks.

https://github.com/BLAKE3-team/BLAKE3

Many people will argue that today's compilers are so smart/optimized that you'd be a fool to try to outsmart the compiler with asm. I'm not 1 of them, but I know some. IMO it's all a bunch of bullshit, there's a goddamn reason all the cryptocurrency mining CPU/GPU code is all hand-written asm. there's a reason blake3 is written in asm ( https://github.com/BLAKE3-team/BLAKE3/blob/master/c/blake3_sse41_x86-64_windows_msvc.asm ) - but the thing is, 99.99% of the time, life is too short to outsmart the compiler (unless you're Alexander Yee)

I have been trying to optimize my code to create a fast hashing function to create and check b3 file integrity but b3sum is way way faster than my aproach, i have been trying to modify my code acordingly to https://github.com/BLAKE3-team/BLAKE3/blob/master/b3sum/src/main.rs with no luck, so if anyone can give me some tips/clues on how to achieve better speeds it would be incredible. Thx!!

BLAKE3 might be faster than KangarooTwelve and is also an XOF. It doesn't have the benefit of getting a working RFC draft proposal however.

You're saying the hash speed is 133 kB/s? That's extremely slow, for example BLAKE3 achieves 6.8 GB/s which is over 50000 times faster. Nobody wants to use such a slow hash function.

Try this one if you want a smaller, and particularly interesting crate: https://github.com/BLAKE3-team/BLAKE3

blake3 is a cryptographic hashing function, which is used during plotting's "forward propagation" step

BLAKE3 claims to be faster and more secure than both MD5 and SHA1.

More information: https://github.com/BLAKE3-team/BLAKE3

> GPU Hash Table?

How bad would performance have suffered if you sha256'd the lines to build the map? I'm going to guess "badly"?

Maybe something like this in CUDA: https://github.com/Cyan4973/xxHash ?

Examples of fast, simple hashes that are independent enough includes murmur, xxHash, Fowler–Noll–Vo hash function and many others

If you're only using the hash for non-cryptographic applications, there are much faster hashes: https://github.com/Cyan4973/xxHash

I found the benchmark result of AMD ryzen 5950X

But what's the catch, you're thinking ? Well, it is a bit slower than its counterparts when it comes to deserializing (and marginally faster for serializing). To achieve smaller footprint, it uses a few tricks and notably a custom hash table to deduplicate strings. This comes at a cost of course (even when featuring xxHash to speed things up), but keeps the slowdown reasonable (I think).

Non cryptographic hashes has collisions, for example, assume you having content like "abcdefg" which hashed value is "123", in case of weak hash algorithm some other content like "abcdefZ" can also have a hash "123" which basically means such hash function is failed to be unique fingerprint of particular content. BLAKE3 for example can do 6-7Gb/s which make it pretty fast and secure. If your requirement accepts collision with defined error rate, I would advise you to take a look at XXH3 if you need very snappy hash algorithm, which can run at pace or RAM access (30GB/s+), but again, run tests at particular equipment you targeting, may be AES hardware accelerated MeowHash will serve you better.