longhorn VS metallb

You can use node selector rules to force Longhorn to use some nodes for storage. You can also add taints to create dedicated storage nodes. Please see the GitHub link listed below. https://github.com/longhorn/longhorn/issues/1633

In Longhorn GUI the volume shows as Detached. It would attach if I do is manually but that makes no impact to the Pod. I've seen people having similar issues in 2614 and have tried with create-default-disk-on-labeled-nodes but that does not seem to work the way I would like to. If I deploy without nodeSelector and disable scheduling on worker nodes it works fine.

rook/longhorn for distributed storage

Longhorn is exactly what you're looking for. It's easy to set up and backs up to NFS.

More than performance, I am worried about security. They don't seem to have considered it at all when building it: https://github.com/longhorn/longhorn/issues/1983

Probably not a good recommendation until this hole is plugged.

They've also got Longhorn, a distributed container-attached storage solution that's very simple to understand and easy to deploy. Performance is another thing but that's the same with all of the general networked storage solutions (Ceph included).

Rancher's got a well deserved good impression in my mind, though early on I avoided it since it seemed like they were building a walled garden.

[0]: https://github.com/longhorn/longhorn

All of their projects have been fully open, monetized with enterprise support. They straight up fully donated their storage project to the CNCF. They've been good stewards of all the tools so far with ones like k3s greatly benefitting the community. And have put a decent amount of effort into putting resources supporting issues on even non-paid issues on the GitHub projects.

Also just in case the above wasn't enough I'm probably actually going to run Longhorn because it's just SO much easier than Ceph to manage and gives me the availability for a ~40% perf loss AFAIK.

Longhorn for Kubernetes storage. It lets me automatically provision PersistentVolumes and snapshot/back them up to S3. I've read that other things such as Ceph and Rook are more stable and I've definitely been bit by some non-critical bugs, but the project is very active and bugs are squashed quickly (though they don't release very frequently). This is the gem I can't live without, being able to snapshot, backup and restore things like my Plex volume has saved me at least a dozen times already.

Another consideration is using something like https://longhorn.io/, it's a bit like ceph, and supports mirroring over two nodes

Are you working with on-premise compute resources or public-cloud based ones? In the latter case, I'd like to point out the need to choose and deploy an Ingress Controller behind which deploy the PHP-powered sites (to minimize the need for public IPs). If the former, you can easily go with MetaLB https://metallb.universe.tf/ to expose each "site" directly, but an Ingress Controller like Kong, Traefik v2 or others could be however handy to accomodate specific needs (OAuth/JWT to access one or more of these, for example). The PHP-enabled website could be deployed with a "multi-container" Pod/Deployment setup as, for example, described here https://matthewpalmer.net/kubernetes-app-developer/articles/php-fpm-nginx-kubernetes.html

Oh. I see, then you probably need to deploy something like MetalLB that satisfies the request for a load balancer

This is a long standing project (Replace the MetalLB configmap with k8s custom resources), it was suggested by the original maintainer pretty much at the beginning of the project.

metalLB to manage bare-metal LoadBalancer services - WIP - Only L2 configuration can be set-up via playbook.

We reused the LB as much as possible to avoid the BGP thing. There's a thing called MetalLB designed around that though.

https://metallb.universe.tf/

That's neat. Looks like metallb uses it too according to go.mod

My latest attempt was MetalLB. Even though I didn't manage to make it work, it bound port 8080 on my machine: none of my other regular Spring demos could work.

I use metalLB as my load balancer because it is software based and any of the nodes can respond to the ip related to the deployment. https://metallb.universe.tf/

If you just need balacing k3s services like controle-plane or ingresses you can use kube-vip with metallb.