ebpf_exporter VS smokeping_prober

There's also the ebpf_exporter which has modules to let you get per-device IO latency data.

For example, we alert when the database starts to get slow, because there's real user impact to slow database queries. We track disk IO latency, but it's for debugging after we get alerted by database alerts. Because database problems could be so many different things. Query plans changing, CPU saturation, etc. Slow reads/writes is just one cause of database problems.

There also a prometheus exporter for the data if you have prom set up. https://github.com/cloudflare/ebpf_exporter

If you have Linux, there is the ebpf_exporter. We use it to monitor real-time disk IO latency, with basically no overhead to the system.

My current go-to to start with storage monitoring is the ebpf_exporter. It allows you to wrap the underlying kernel functions with timing wrappers, which allows you to gather detailed data on every IO that happens. Gather that up into a histogram, and ship it to a monitoring system for visualization.

Check out the smokeping_prober. It generates high resolution histogram data that you can use to compare different paths.

I wrote this smokeping tool to monitor. I also use unpoller

I wrote this tool to monitor my ISP connectivity.

I am thinking this might be the exporter to use: https://github.com/SuperQ/smokeping_prober At the bottom of that page it says

I wrote a smokeping_prober for Prometheus.

For some stuff, I do end-to-end probin with a tool I wrote.

Smokeping Prober

I wrote a "better smokeping" tool that produces much more detailed data than IP SLA.

I wrote my own variation of smokeping. It currently only does ICMP and UDP probing, but I've thought about adding things like TCP, HTTP, DNS, etc.