py-inventa VS rkvdns

- Inventa for Python: https://github.com/adalkiran/py-inventa - https://pypi.org/project/inventa

Agree with the article enough that I did something about it which I call "Poor Fred's SIEM". The heart of it is a DNS proxy for Redis (https://github.com/m3047/rkvdns). However it's not targeted at environments where everything is in a "bubble" such that there are no ingress / egress costs. (Lookin' at you, Cloud.) Furthermore "control plane" is an important concept, and it's well understood in the industrial control world as the Purdue Model.

From a systems standpoint do you need to have all resources stored centrally in order to do centralized reporting? No, of course not. Admittedly it's handy if bandwidth and storage are free. The alternative is distributed storage, with or without summarization at the edge (and aggregating from distributed storage for reporting).

Having it distributed does raise access issues: access needs to be controlled, and management of access needs to be managed. Philosophically the Cloud solutions sell centralized management, but federation is a perfectly viable option. The choice is largely dictated by organizational structure not technology.

There is also a difference between diagnostic and evaluative indicators. Trying to evaluate from diagnostics causes fatigue because humans aren't built that way; evaluatives can and should be built from diagnostics. Diagnostics can't be built from evaluatives.

The logging/telemetry stack that I propose is:

1) Ephemeral logging at the limits of whatever observability you can build. E.g.: systemd journal with a small backing store, similar to a ring buffer.

2) Your compliance framework may require shipping some classes of events off of the local host, but I don't think any of them require shipping it to the cloud.

3) Build evaluatives locally in Redis.

4) Use DNS to query those evaluatives from elsewhere for ad hoc as well as historical purposes. This could be a centralized location or it could be true federation where each site accesses all other site's evaluatives.

I wouldn't put Redis on the internet, but I don't worry too much about DNS; and there are well-understood ways of securing DNS from tampering, unauthorized access, and even observation. By the way, DNS will handle hundreds or thousands of queries per second you just have to build for it.

DNS is everywhere. It offers massive caching potential. It is well-understood and there are multiple, interoperating, implementations. It is a data diode, which is to say read-only, which can simplify some risk assessments related to attack surface.

The README for this project may explain some of the general benefits (my project): https://github.com/m3047/rkvdns

HTTP is everywhere. I think SOAP sucks, but it's everywhere and it works well enough. (For that matter XHR sucks, but even I use it occasionally.)

(This is not the first time DNS Toys has shown up on Hacker News, there's a link in another comment.)

Create your own adventure, all you need is python + dnspython + dns redis, and a Redis DB: https://github.com/m3047/rkvdns

When you start using DNS as a generalized key/value store, there are some tuning / optimizations to be aware of:

* Production grade caching / recursing servers retry aggressively. There is debouncing in this implementation.

* Tune your EDNS packet size (in your caching server) to make sure you aren't triggering retries unnecessarily. (And frags are bad and Francisco Franco is still dead.)

* Empty non-terminals are rare enough in "happy eyeballs" use that (de) optimizations in the name of things like privacy are known to happen. You should contemplate disabling qname minimization if that's something the caching server does.

Looking for an active project or actual production deployment with thousands as opposed to millions of keys, with lots of reads and writes. I've got a way to offload most of the reads to DNS infrastructure which is very efficient for caching reads.

As far as architectural limits, no KEYS operation should generate more than roughly 64K of data and keys / values themselves should be relatively short, 64-128 bytes with a hard limit of 255 bytes per value. (Data is not restricted to e.g. ASCII, as the DNS says "any octet".)

This is a read only solution, so it implicitly protects against malicious writes; and the Redis database only has to be accessible to the translation layer. DNS features provide control over TTLs, WAF-like control over access to specific keys, restriction by IP address and "API key" (TSIG, which also provides integrity guarantees). Encryption, if necessary, can be implemented with TLS (including endpoint termination with Nginx) or VPN.

A federated architecture with multiple Redis instances is absolutely in scope, and indeed one of the "just in time collection from the edge" scenarios where the DNS particularly shines.

I'll provide 20 hours of assistance for free, what I want in return is a commitment to write up and publish a case study.

This is the tool: https://github.com/m3047/rkvdns and it would be deployed in conjunction with instances of BIND (https://www.isc.org/bind) to provide the advanced features.

Reach out to me at [email protected] or the email address referenced in the project README for paid support (bonus points for the latter) and mention you saw this post. Feel free to ask general questions below.

> his father started working with programmable logic controllers (PLCs), and his interest in these controllers led him to purchase a Z80 processor board and start programming on it

In the PLC world there is a notion of "tags". These are values pushed or poked by PLCs with some kind of external (networked) access. They are essentially network-wide global variables. If you think about it for a little while this looks like key+value store. There are some common operations with tags: on/off, counters, bit fields, along with static/updating values. There is a notion that if a tag hasn't been updated in a while it is "stale" or unreliable.

Although my POC deployment of RKVDNS (DNS Proxy for Redis https://github.com/m3047/rkvdns/) revolves around SecOps / DevOps / DevSecOps, my horizon is SCADA and rationalizing the federation (edge) vs centralization (cloud) dilemma for observables, leveraging the reality that these days even (Purdue) Level 1 has DNS.

All your counters, sets and lists you've got in Redis, with the equivalent of a WAF: https://github.com/m3047/rkvdns