autometrics-rs VS criterion.rs

I’ve been working on an open source project built on OpenTelemetry and Prometheus client libraries (https://autometrics.dev). The DX of the OTel libraries is pretty painful in all the languages we’ve used. Granted, we’re using them for metrics while traces are more common, but still.

I think the fundamental issue for the DX is that it’s trying to do everything everyone might want out of all of the observability signals. That’s a useful and laudable goal but it means that everything is configurable and relatively difficult to use.

If you go with Prometheus, we’re building Autometrics to make producing and querying metrics easier for developers. It makes it trivial to instrument functions to track the request rate, error rate, and latency and then writes PromQL for you. The Autometrics libraries are thin layers on top of existing Prometheus and OpenTelemetry libraries.

Autometrics is an open source observability framework that makes it trivial to add useful metrics to your code and writes Prometheus queries for you to help you understand the data. This feature shows the power that comes from pairing code instrumentation with automatically writing queries (and the queries it writes are a whole lot more complicated than what you'd want to write by hand!).

How do people tend to get metrics off of embedded devices?

I’m working on https://github.com/autometrics-dev/autometrics-rs and people asked whether it could be used in embedded contexts but I wasn’t sure how you’d hook up the device to something like Prometheus.

I just opened these two issues for [supporting `prometheus-client`](https://github.com/fiberplane/autometrics-rs/issues/25) and another for [supporting exemplars](https://github.com/fiberplane/autometrics-rs/issues/26).

The three popular options for benchmarking in Rust are: libtest bench, Criterion, and Iai.

Holy crap. I was just writing a blog to complain about the state of Rust benchmarking and I think this might address most of my points. The biggest one is the ability to have benchmarks collocated within the library like tests which is the biggest annoyance.

It’s also nice to see that it can report multiple counters in parallel. I put up a similar feature[1] for criterion recently but I fear the project isn’t being maintained anymore…

Haven’t looked deeply into divan yet but the other requirements I have for criterion’s power is to run tests with statistical guarantees on the results, terminate quickly when statistical significance is reached (—quick), provide a comparison of the delta from a previous benchmark, and to run async code. Wonder how this stacks up.

[1] https://github.com/bheisler/criterion.rs/pull/722

criterion for benchmarks,

For an extra optimization, you might look into techniques like branchless code, turning array of structs into struct of arrays for better cache utilization/lowering branch mispredictions. There is many talks on YouTube how to actually measure and improve performance of native code. And for rust there is a criterion.rs for benchmarking.

https://github.com/bheisler/criterion.rs is good for tests like that. It will give you much more than a single number and handle things like outliers. This makes identifying noisy tests simpler.

I have tried many ways but I think is not possible. I put it on https://github.com/bheisler/criterion.rs/issues/631.

Criterion is still the gold standard.

Pros for Criterion over the stdlib: https://github.com/bheisler/criterion.rs#features

Downsides of Criterion: https://bheisler.github.io/criterion.rs/book/user_guide/know...