longhorn VS zfs-localpv

You can use node selector rules to force Longhorn to use some nodes for storage. You can also add taints to create dedicated storage nodes. Please see the GitHub link listed below. https://github.com/longhorn/longhorn/issues/1633

In Longhorn GUI the volume shows as Detached. It would attach if I do is manually but that makes no impact to the Pod. I've seen people having similar issues in 2614 and have tried with create-default-disk-on-labeled-nodes but that does not seem to work the way I would like to. If I deploy without nodeSelector and disable scheduling on worker nodes it works fine.

rook/longhorn for distributed storage

Longhorn is exactly what you're looking for. It's easy to set up and backs up to NFS.

More than performance, I am worried about security. They don't seem to have considered it at all when building it: https://github.com/longhorn/longhorn/issues/1983

Probably not a good recommendation until this hole is plugged.

They've also got Longhorn, a distributed container-attached storage solution that's very simple to understand and easy to deploy. Performance is another thing but that's the same with all of the general networked storage solutions (Ceph included).

Rancher's got a well deserved good impression in my mind, though early on I avoided it since it seemed like they were building a walled garden.

[0]: https://github.com/longhorn/longhorn

All of their projects have been fully open, monetized with enterprise support. They straight up fully donated their storage project to the CNCF. They've been good stewards of all the tools so far with ones like k3s greatly benefitting the community. And have put a decent amount of effort into putting resources supporting issues on even non-paid issues on the GitHub projects.

Also just in case the above wasn't enough I'm probably actually going to run Longhorn because it's just SO much easier than Ceph to manage and gives me the availability for a ~40% perf loss AFAIK.

Longhorn for Kubernetes storage. It lets me automatically provision PersistentVolumes and snapshot/back them up to S3. I've read that other things such as Ceph and Rook are more stable and I've definitely been bit by some non-critical bugs, but the project is very active and bugs are squashed quickly (though they don't release very frequently). This is the gem I can't live without, being able to snapshot, backup and restore things like my Plex volume has saved me at least a dozen times already.

Another consideration is using something like https://longhorn.io/, it's a bit like ceph, and supports mirroring over two nodes

Ideas from my kubernetes experience: * Cert-Manager is very popular and almost a must-have if you terminate SSL inside the cluster * Backups using velero * A dashboard/UI is actually very helpful to quickly browse resources, client tools like k9s are fine too * Secret: Management: Bitnami Sealed Secrets is the second big project in that space * I would add Loki to aggregate Logs * Never heard of ory. Usually I see (dex)[https://dexidp.io/] or keycloak used for Authentication * I like to run OpenEBS as in-cluster storage. * Istio isn't compatible with the upcomming ServiceMeshInterface (i think), so the trend seem to go toward Linkerd * Some Operator to deploy your favorite Database, is also a nice learning exercise.

OpenEBS ZFS LocalPV (declared stable), used widely for production workloads that need direct and resilient storage has added new capabilities like:

So I'm in the process of converting every node in my cluster to ZFS (RAID1 - a single mirror zpool) (powered by OpenEBS's ZFS LocalPV for the k8s heads out there). I'm going to try running Ceph w/ bluestore but doing something that would normally make absolutely no sense -- disabling the checksumming features of Ceph.