nhi VS ebpf_exporter

Check out nhi.

You may find this interesting.

Keep in mind that nhi is in early alpha stage and I plan to implement more features (for instance that: https://github.com/strang1ato/nhi/issues/8)

Not yet, but I plan to add that feature https://github.com/strang1ato/nhi/issues/8

I created a tool which automatically captures every potentially useful information about each executed command inside bash/zsh, and delivers powerful querying mechanism.

These features allow retrievement of commands (with outputs) executed in the past and whole shell sessions, as well as every other useful information in a convenient way.

nhi github: https://github.com/strang1ato/nhi

There's also the ebpf_exporter which has modules to let you get per-device IO latency data.

For example, we alert when the database starts to get slow, because there's real user impact to slow database queries. We track disk IO latency, but it's for debugging after we get alerted by database alerts. Because database problems could be so many different things. Query plans changing, CPU saturation, etc. Slow reads/writes is just one cause of database problems.

There also a prometheus exporter for the data if you have prom set up. https://github.com/cloudflare/ebpf_exporter

If you have Linux, there is the ebpf_exporter. We use it to monitor real-time disk IO latency, with basically no overhead to the system.

My current go-to to start with storage monitoring is the ebpf_exporter. It allows you to wrap the underlying kernel functions with timing wrappers, which allows you to gather detailed data on every IO that happens. Gather that up into a histogram, and ship it to a monitoring system for visualization.