xtemplate VS xxHash

I've also been dissatisfied with http caching not utilizing content hashes enough. If you're using server side templating one issue is that it's not efficient to calculate the hash while you're running the template, it would need to be precalculated to be efficient enough to use.

So I wrote https://github.com/infogulch/xtemplate to scan all assets at startup to precalculate the hash for templates that use it, and if a request comes in with a query parameter ?hash=sha384-xyz and it matches then it gives it a 1 year immutable Cache-Control header automatically. If a file x.ext has a matching x.ext.gz/x.ext.zst/x.ext.br file then (after hashing the content to make sure it matches) client requests that support it are sent a compressed version streamed directly from disk with sendfile2. I call this "Optimal asset serving" (a bit bold perhaps).

Yes, that would be useful. Basically, what you have in https://github.com/infogulch/caddy-xtemplate/blob/master/tem... - but without Caddy dependencies and with a way to use any router library (I use gorilla/mux).

I would make the templates library take a callback function for every public template: func(method, path string, template).

Then the user could add any custom logic to register the route handler and execute the template with any application specific inputs.

> GPU Hash Table?

How bad would performance have suffered if you sha256'd the lines to build the map? I'm going to guess "badly"?

Maybe something like this in CUDA: https://github.com/Cyan4973/xxHash ?

Examples of fast, simple hashes that are independent enough includes murmur, xxHash, Fowler–Noll–Vo hash function and many others

If you're only using the hash for non-cryptographic applications, there are much faster hashes: https://github.com/Cyan4973/xxHash

I found the benchmark result of AMD ryzen 5950X

But what's the catch, you're thinking ? Well, it is a bit slower than its counterparts when it comes to deserializing (and marginally faster for serializing). To achieve smaller footprint, it uses a few tricks and notably a custom hash table to deduplicate strings. This comes at a cost of course (even when featuring xxHash to speed things up), but keeps the slowdown reasonable (I think).

Non cryptographic hashes has collisions, for example, assume you having content like "abcdefg" which hashed value is "123", in case of weak hash algorithm some other content like "abcdefZ" can also have a hash "123" which basically means such hash function is failed to be unique fingerprint of particular content. BLAKE3 for example can do 6-7Gb/s which make it pretty fast and secure. If your requirement accepts collision with defined error rate, I would advise you to take a look at XXH3 if you need very snappy hash algorithm, which can run at pace or RAM access (30GB/s+), but again, run tests at particular equipment you targeting, may be AES hardware accelerated MeowHash will serve you better.