criterion.rs VS sccache

The three popular options for benchmarking in Rust are: libtest bench, Criterion, and Iai.

Holy crap. I was just writing a blog to complain about the state of Rust benchmarking and I think this might address most of my points. The biggest one is the ability to have benchmarks collocated within the library like tests which is the biggest annoyance.

It’s also nice to see that it can report multiple counters in parallel. I put up a similar feature[1] for criterion recently but I fear the project isn’t being maintained anymore…

Haven’t looked deeply into divan yet but the other requirements I have for criterion’s power is to run tests with statistical guarantees on the results, terminate quickly when statistical significance is reached (—quick), provide a comparison of the delta from a previous benchmark, and to run async code. Wonder how this stacks up.

[1] https://github.com/bheisler/criterion.rs/pull/722

criterion for benchmarks,

For an extra optimization, you might look into techniques like branchless code, turning array of structs into struct of arrays for better cache utilization/lowering branch mispredictions. There is many talks on YouTube how to actually measure and improve performance of native code. And for rust there is a criterion.rs for benchmarking.

https://github.com/bheisler/criterion.rs is good for tests like that. It will give you much more than a single number and handle things like outliers. This makes identifying noisy tests simpler.

I have tried many ways but I think is not possible. I put it on https://github.com/bheisler/criterion.rs/issues/631.

Criterion is still the gold standard.

Pros for Criterion over the stdlib: https://github.com/bheisler/criterion.rs#features

Downsides of Criterion: https://bheisler.github.io/criterion.rs/book/user_guide/know...

Use icecream or sccache. sccache supports distributed builds.

https://github.com/mozilla/sccache/blob/main/docs/Distribute...

Worth noting that the first commit in sccache git repository was in 2014 (https://github.com/mozilla/sccache/commit/115016e0a83b290dc2...). So I suppose that what "happened" happened waay back.

If you have lots of shared dependencies, maybe try sccache?

I'm going to set up sccache [0] to use it tomorrow. We use MSVC, so EFS is off the cards.

[0] https://github.com/mozilla/sccache/blob/main/docs/S3.md

I think the primary benefit of pre-built procmacros will be for build servers which don't use a persistent cache (like sccache), since they have to compile all dependencies every time. But IMO improved support for persistent caches would be a better investment compared to adding support for pre-built procmacros.

Checkout https://github.com/mozilla/sccache

https://github.com/mozilla/sccache is another option which addresses the use cases of both icecream and ccache (and also supports Rust, and cloud storage of artifacts, if those are useful for you)

That being said a compilation cache, eg the de-facto standard for Rust: sccache (https://github.com/mozilla/sccache) will help to compile and store some of the build artifacts centralized - still for each crate version + build profile (RUSTFLAGS) combination.