Our great sponsors
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
Agree with the article enough that I did something about it which I call "Poor Fred's SIEM". The heart of it is a DNS proxy for Redis (https://github.com/m3047/rkvdns). However it's not targeted at environments where everything is in a "bubble" such that there are no ingress / egress costs. (Lookin' at you, Cloud.) Furthermore "control plane" is an important concept, and it's well understood in the industrial control world as the Purdue Model.
From a systems standpoint do you need to have all resources stored centrally in order to do centralized reporting? No, of course not. Admittedly it's handy if bandwidth and storage are free. The alternative is distributed storage, with or without summarization at the edge (and aggregating from distributed storage for reporting).
Having it distributed does raise access issues: access needs to be controlled, and management of access needs to be managed. Philosophically the Cloud solutions sell centralized management, but federation is a perfectly viable option. The choice is largely dictated by organizational structure not technology.
There is also a difference between diagnostic and evaluative indicators. Trying to evaluate from diagnostics causes fatigue because humans aren't built that way; evaluatives can and should be built from diagnostics. Diagnostics can't be built from evaluatives.
The logging/telemetry stack that I propose is:
1) Ephemeral logging at the limits of whatever observability you can build. E.g.: systemd journal with a small backing store, similar to a ring buffer.
2) Your compliance framework may require shipping some classes of events off of the local host, but I don't think any of them require shipping it to the cloud.
3) Build evaluatives locally in Redis.
4) Use DNS to query those evaluatives from elsewhere for ad hoc as well as historical purposes. This could be a centralized location or it could be true federation where each site accesses all other site's evaluatives.
I wouldn't put Redis on the internet, but I don't worry too much about DNS; and there are well-understood ways of securing DNS from tampering, unauthorized access, and even observation. By the way, DNS will handle hundreds or thousands of queries per second you just have to build for it.