longhorn VS loki

You can use node selector rules to force Longhorn to use some nodes for storage. You can also add taints to create dedicated storage nodes. Please see the GitHub link listed below. https://github.com/longhorn/longhorn/issues/1633

In Longhorn GUI the volume shows as Detached. It would attach if I do is manually but that makes no impact to the Pod. I've seen people having similar issues in 2614 and have tried with create-default-disk-on-labeled-nodes but that does not seem to work the way I would like to. If I deploy without nodeSelector and disable scheduling on worker nodes it works fine.

rook/longhorn for distributed storage

Longhorn is exactly what you're looking for. It's easy to set up and backs up to NFS.

More than performance, I am worried about security. They don't seem to have considered it at all when building it: https://github.com/longhorn/longhorn/issues/1983

Probably not a good recommendation until this hole is plugged.

They've also got Longhorn, a distributed container-attached storage solution that's very simple to understand and easy to deploy. Performance is another thing but that's the same with all of the general networked storage solutions (Ceph included).

Rancher's got a well deserved good impression in my mind, though early on I avoided it since it seemed like they were building a walled garden.

[0]: https://github.com/longhorn/longhorn

All of their projects have been fully open, monetized with enterprise support. They straight up fully donated their storage project to the CNCF. They've been good stewards of all the tools so far with ones like k3s greatly benefitting the community. And have put a decent amount of effort into putting resources supporting issues on even non-paid issues on the GitHub projects.

Also just in case the above wasn't enough I'm probably actually going to run Longhorn because it's just SO much easier than Ceph to manage and gives me the availability for a ~40% perf loss AFAIK.

Longhorn for Kubernetes storage. It lets me automatically provision PersistentVolumes and snapshot/back them up to S3. I've read that other things such as Ceph and Rook are more stable and I've definitely been bit by some non-critical bugs, but the project is very active and bugs are squashed quickly (though they don't release very frequently). This is the gem I can't live without, being able to snapshot, backup and restore things like my Plex volume has saved me at least a dozen times already.

Another consideration is using something like https://longhorn.io/, it's a bit like ceph, and supports mirroring over two nodes

Jaeger - CNCF Jaeger, a Distributed Tracing Platform Kiali - Kiali project, observability for the Istio service mesh ELK - Elasticsearch, Logstash, Kibana fluentbit - Fast and Lightweight Log processor and forwarder for Linux, BSD and OSX Loki - Like Prometheus, but for logs

Have a look at https://github.com/grafana/loki for example. Based on it and the source for Kubectl, my team has adopted gRPC, and two libs called Cobra and Viper by https://github.com/spf13

Loki

or Grafana, Prometheus and Loki (Grafana has this example using Tanka/ksonnet) is a good Benchmark https://github.com/grafana/loki/blob/main/docs/sources/installation/tanka.md

No, Loki is for logging.