rkvdns_examples VS Netdata

Seems defeatist to me.

1) There has to be a notion that some things are worth acknowledging as "events"; this leads to the idea that what logs contain is indicators of events. It's a fundamentally philosophical notion. It means you need to take the time to decide what constitutes an event. Hearkening to machine learning and pirates, global warming may inversely correlate with pirates but that doesn't imply causation (either way): you can't just throw statistical techniques at data looking for "hits" and think that's significant. Even if you find some indicator as the article notes it could change; so you should identify some canary indicators and event those as well.

2) Which leads to the point about "bug parts": don't rely on a specific rare indicator, or the failure to identify such an indicator. If you find high-reliability indicators great, but look for other indicators which occur more often, that can be counted, and track those. For instance an indicator that e.g. systemd is restarting /something/, and that's happening more or less frequently, and correlates with a performance observable. If it stops reporting at all, you can start with the presumption that something about logging itself changed.

At this point my philosophical disagreement with centralized logging comes to the fore: it's expensive to load stuff into Splunk. I agree, and that's why I disagree with the approach and prefer federation.

You can use the Totalizer Agent (https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) to increment counters in Redis for regex-identified keys. I don't care whether you use RKVDNS to retrieve the data or something else.

In general this evolves to a SIEM-like solution in IT or gets added to the tag menagerie in OT.

If you're focused on "notifications are bad" note that notifications are push, and pull solutions are possible. Tail logs (or journalctl) and post significant events to Redis (https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) for example.

netdata.cloud — Netdata is an open-source tool to collect real-time metrics. It's a growing product and can also be found on GitHub!

Netdata is designed with efficiency, scalability, and flexibility in mind, aiming to address most of the challenges associated with both open-source tools and commercial SaaS offerings.

Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

We run netdata to have a bit of insight into whats happening on the 10+ dedicated servers in Falkenstein. So far we have seen a 1hour_ecc_memory_correctable about once a month. Suddenly we get 1hour_ecc_memory_correctable like every day from different servers. Any ideas why that could be happening?

Documentation and source code of this plugin: https://github.com/netdata/netdata/tree/master/collectors/systemd-journal.plugin

Home Page and source code: https://github.com/netdata/netdata

I started the discussion, and offered a solution too:

https://github.com/netdata/netdata/discussions/16136

hey - I work on ML at Netdata (disclaimer).

We have a big PR open and under review at moment that brings in a lot more logs capabilities: https://github.com/netdata/netdata/pull/13291

We also have some specific logs collectors too - i think in here might be best place to look around at the moment, should take you to the logs part of the integrations section in our demo space (no login needed, sorry for the long horrible url, we adding this section to our docs soon but at moment only lives in the app)

https://app.netdata.cloud/spaces/netdata-demo/rooms/all-node...