longhorn VS loki

For my primary CSI provider, I use Longhorn, and to administrate my clusters, I use the Kubernetes Dashboard. In front of all of my private dashboards, I use Traefik Forward Auth to limit who can access them.

Maybe with https://longhorn.io

For data must be protected, I use Longhorn https://longhorn.io/ which is basically iSCSI/NFS with replication.

Backups are definitely necessary. The problem though is that you cant detect by default if you are backing up corrupted data. By now i found three approaches: 1. Use a filesystem that detects and fixes errors (e. g. btrfs or zfs) 2. Use par2 (parchive) to generate checksums on your files manually 3. I found the following issue on the k8s storage solution I currently use: https://github.com/longhorn/longhorn/issues/3198 For now I am waiting for option 3 getting merged. In the long run I will upgrade my laptop to btrfs and for now use par2. Hope that provides a starting point. Feel free to add your ideas!

You can use the Longhorn iSCSI installer script as an example. https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-iscsi-installation.yaml

Rancher's Longhorn (even though it's only a single-node)

Application storage now: yes, NAS is probably the best way to go about it. I believe k3s uses some kind of "local storage" by default but there's plenty of alternative Container Storage Interfaces (CSI) out there. Longhorn worked well for me and it covers most uses cases I care about: snapshots, backups, and replication. I don't know about any "NAS providers" though, and once again there will be performance drawbacks with having your storage that externalized.

For me longhorn works really well as my storage backend.

Longhorn is a lightweight, reliable, and powerful distributed block storage system for Kubernetes.

You mentioned storage... That's always tricky but provisioning persistent volumes with https://longhorn.io is really simple... It's also easy to setup, the helm chart is wonderful

Like all the top posts here say, you're going to want Loki. Probably Promtail to collect logs and push them to Loki (although any log collector works..I've also used Fluentd, Fluent-bit, and logstash). Setup a data source in Grafana and it works great. I install all of them from Grafana's own helm charts: https://github.com/grafana/loki/tree/main/production/helm

> Is there an open source version of splunk I can modify?

https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.

OP specifically mentioned APM, which is what maps to Tempo. If you wanted to go full Grafana for metrics and logs too, you can add in Mimir and Loki

I notice that Loki's treadme says it doesn't index log content. I don't know at what scale that becomes an issue. Have you had any performance concerns?

Similarly, fluentd is much more like NoSQL as a datastore than anything else. Prometheus is fundamentally a TSDB, and Loki is moving that direction (it's currently boltdb, which is also just a k:v store).

I don't quite understand the window-logic but for logs there are things like Logstash, Loki, Graylog and probably many others that are meant to handle logs.