word_frequency_nim VS raikv

Thanks for doing a nim version! I've been learning nim, and am very much a beginner, so I thought I'd tackle a simple version [0]. I'd appreciate any critiques... I don't know how idiomatic it is.

Honestly, the optimized version you created is kind of opaque; I don't want people to think that that's what "typical" nim looks like.

I'll do a pull request on the project for mine as simple, yours as optimized, if you don't mind.

[0]: https://github.com/csterritt/word_frequency_nim/blob/master/...

I have a graph for this:

https://github.com/raitechnology/raikv/blob/master/graph/mt_...

The CPU in this case is a Threadripper 3970x, 32 cores, 64 SMT.

My experience is this: When the L3 cache is effective, then the memory latency hiding via memory prefetch works well across SMT threads. If the hashtable load requires a chain walk, the SMT latency hiding is less effective because the calculated prefetch location is not the actual hit. I couldn't get prefetching multiple slots as the load increased to be as effective as prefetching a single slot.

Amusingly, I've done a multi-threaded version of the word counting program in order to test a shm kv store. I needed benchmark that created a lot of cross thread concurrent accesses to keys and I found a blog about this test. My version has serious constraints though, you have to create a shared memory map with enough space to hold all of the keys beforehand, as it doesn't resize the shm kv map as it runs.

This is the source for it:

https://github.com/raitechnology/raikv/blob/master/test/ctes...

The speedup of the multi-threaded version vs the single-threaded version is about linear. The single threaded version uses 2 threads, one to read stdin and one to hash the keys, the 16 threaded version uses one thread to read, 16 to hash.

$ time ctest -t 1 < ~/data/enwiki-p10p30303