bpftrace VS ebpf_exporter

In this blog, I will present secimport — a toolkit for creating and running sandboxed applications in Python that utilizes eBPF (bpftrace) to secure Python runtimes.

I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux:

Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx).

Associate PIDs that open those files with their .desktop entries[1]

Store results in some database like sqlite3.[2]

Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3].

I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper

[0]: https://github.com/iovisor/bpftrace

I develop a python tool that allows you to trace a Python process line by line or the functions' entries and returns. It uses USDT(User Statically-Defined Tracing) probes with bpftrace.

bpftrace and/or bpfcc-tools can also be useful (dpkg -L bpftrace to see available tools). You can monitor files being opened/written at kernel level (opensnoop*, filelife*, filetop*), connections being established (tcp*bpfcc), etc.

Similar to this method is bpftrace: https://github.com/iovisor/bpftrace/blob/master/tools/execsnoop.bt

uname -a Linux ying 5.18.5-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 16 14:51:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

There's also the ebpf_exporter which has modules to let you get per-device IO latency data.

For example, we alert when the database starts to get slow, because there's real user impact to slow database queries. We track disk IO latency, but it's for debugging after we get alerted by database alerts. Because database problems could be so many different things. Query plans changing, CPU saturation, etc. Slow reads/writes is just one cause of database problems.

There also a prometheus exporter for the data if you have prom set up. https://github.com/cloudflare/ebpf_exporter

If you have Linux, there is the ebpf_exporter. We use it to monitor real-time disk IO latency, with basically no overhead to the system.

My current go-to to start with storage monitoring is the ebpf_exporter. It allows you to wrap the underlying kernel functions with timing wrappers, which allows you to gather detailed data on every IO that happens. Gather that up into a histogram, and ship it to a monitoring system for visualization.