countwords VS llfio

"dang, I didn't know that was 50x faster than the idiomatic way" or "hey, I didn't know that this implementation in the stdlib prioritized this over that and made this so slow, that's interesting" -- .e.g, there's some kinda neat language details to be found in something like Ben Hoyt's community word count benchmarks repo and 'simple' vs 'optimal' code: https://github.com/benhoyt/countwords

It benchmarks programs that count the total number of unique words in some input. It's not exactly equivalent to your problem, but it's similarish. All of the programs used some kind of hash map for lookups, but I contributed a program that used a trie. Its performance in my experience varies depending on the CPU interestingly enough. On my old CPU (i7-6900K) it was a little slower, but on my new cpu (i9-12900KS) it was faster.

Why not read the source code? :-)

I wrote comments explaining things: https://github.com/benhoyt/countwords/blob/8553c8f600c40a462...

My own code example of a drastic speed up (~25%) simply replacing a couple of for loops with iters: https://github.com/benhoyt/countwords/pull/115

And yet Go was faster than Rust in a simple app that count words: https://benhoyt.com/writings/count-words/

> but the performance profiles & characteristics that we must know about in order to make a choice on which tool to use. And it shouldn't be that each user has to figure it out on their own, dig into PR's or whatever.

That's an interesting take – I like the idea of a catalog of standard tasks with implementations in several languages as well as their performance characteristics. I suppose Rosetta Code gets the ball rolling with this, but it's missing some performance metrics. It reminds me of [Ben Hoyt's piece](https://benhoyt.com/writings/count-words/) on counting unique words in the KJV Bible in different languages.

This article shows that map lookups can be optimized by using the (unintuitive) pattern:

The reference library can be found at https://ned14.github.io/llfio/

Thanks for the suggestions, which I have transposed into https://github.com/ned14/llfio/issues/106

There was an effort to get an afio library accepted into boost in the past. I believe the most current work on that library is happening here nowadays : https://github.com/ned14/llfio I'm not sure if it is considered production-ready or not. But I couldn't see any mention of it in the replies so I figured I would fix that!

It has an implementation: LLFIO

I'm a fan of Interprocess, used it for over a decade. But for mmapping I've switched to LLFIO and recommend it highly. (Plugging so Niall doesn't have to.)

Since that post, I have the reference implementation library very nearly passing its test suite https://github.com/ned14/llfio/pull/89. Once it's done I'll start very slowly writing its proposal paper for WG21 SG4. Should land before this summer.

FYI it doesn't solve quite what you're solving, but I've been careful to ensure https://github.com/ned14/llfio works well on Freestanding and < 64 Kb microcontrollers and I know Victor has been careful to ensure a good subset of std::format could work well on embedded. In other words, the i/o story for embedded C++ may improve greatly in the next few years.

If you want stronger guarantees than iostreams can give you, either use the OS-specific calls or a wrapper of said calls (e.g. https://github.com/ned14/llfio, disclaimer I'm the owner of that). Note that even in LLFIO, there is no concept of "seek to the end" because that's racy so we don't implement that. All you get is atomic append, otherwise you're on your own to coordinate what "end of file" means.