criterion.rs VS hyperfine

The three popular options for benchmarking in Rust are: libtest bench, Criterion, and Iai.

Holy crap. I was just writing a blog to complain about the state of Rust benchmarking and I think this might address most of my points. The biggest one is the ability to have benchmarks collocated within the library like tests which is the biggest annoyance.

It’s also nice to see that it can report multiple counters in parallel. I put up a similar feature[1] for criterion recently but I fear the project isn’t being maintained anymore…

Haven’t looked deeply into divan yet but the other requirements I have for criterion’s power is to run tests with statistical guarantees on the results, terminate quickly when statistical significance is reached (—quick), provide a comparison of the delta from a previous benchmark, and to run async code. Wonder how this stacks up.

[1] https://github.com/bheisler/criterion.rs/pull/722

criterion for benchmarks,

For an extra optimization, you might look into techniques like branchless code, turning array of structs into struct of arrays for better cache utilization/lowering branch mispredictions. There is many talks on YouTube how to actually measure and improve performance of native code. And for rust there is a criterion.rs for benchmarking.

https://github.com/bheisler/criterion.rs is good for tests like that. It will give you much more than a single number and handle things like outliers. This makes identifying noisy tests simpler.

I have tried many ways but I think is not possible. I put it on https://github.com/bheisler/criterion.rs/issues/631.

Criterion is still the gold standard.

Pros for Criterion over the stdlib: https://github.com/bheisler/criterion.rs#features

Downsides of Criterion: https://bheisler.github.io/criterion.rs/book/user_guide/know...

Check out the official hyperfine Github repo

And then I used hyperfine to run the benchmarks on my MacBook Pro 14 M2 Max, and here are the results:

Search "benchmarking tools for linux" and decide that hyperfine is good for what I'm doing. Run Jennifer's new python script against my refactored perl and find that the python is 1.26 times faster for k=3 and 1.47 times faster for k=4. For the Covid-19 sequence, these are both on the order of hundreds of milliseconds.

> It is very possible to write sub 100ms procedures in TS, […]

I will not disagree with this statement because I don’t have a way to test inshellisense right now. Could you (or anyone with a working Node + NPM installation) please install inshellisense and post the actual numbers? Perhaps using a tool like hyperfine (https://github.com/sharkdp/hyperfine).

Yeah, while it's not as thorough as these tools, the method is at least reproducible and sane, and with ~10 or so samples, you get an interval with a nice confidence.

Another through method will be hyperfine[0], yet I wanted to provide a method which requires no installation and can be done in a whim, without jumps and hoops, with the tools already at hand.

[0]: https://github.com/sharkdp/hyperfine

That is native and inbuild but I would suggest below options instead 1. Using lazy's Profile tab instead https://github.com/folke/lazy.nvim 2. Using a dedicated plugin to do this https://github.com/dstein64/vim-startuptime. 3. Using an external program hyperfine is one that I use https://github.com/sharkdp/hyperfine

Fair enough, although might I recommend using hyperfine for your testing? ;p