Our great sponsors
-
I really like this article just for the straightforwardness of the setup. Pets not cattle should be the home server mantra.
My setup is not quite as simple. I have one homeprod server running Proxmox with a number of single task VMs and LXCs. A task, for my purposes, is a set of one or more related services. So I have an internal proxy VM that also runs my dashboard. I have a media VM that runs the *arrs. I have an LXC that runs Jellyfin (GPU pass through is easier with LXC). A VM running Home Assistant OS. Etcetera.
Most of these VMs are running Docker on top of Alpine and a silly container management scheme I've cooked up[1]. I've found this setup really easy to wrap my head around, vs docker swarm or k8s or what have you. I'm even in the process of stripping dokku out of my stack in favor of this setup.
-
Moby
The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
So, for context, my experience is limited to trying to get a MariaDB Galera cluster running. Specifically using the Bitnami image. So my issues might not apply to every single stateful app out there. I'm also running all of this on a vsphere in our own data center. Not in the cloud.
Swarm does not support dependencies between services. See [0]. It also does not support deploying replicas one at a time. See [1] where I'm asking for that support.
In the case of Galera, you need a master node to be up and running before you add any new nodes. I'm pretty sure that when you're initiating any kind of stateful clustered app, you'd want to start with one node at a time to be safe. You can't do that in Swarm using a replicated service. All replicas start at the same time.
Using a service per instance might work, but you need to be sure you have storage figured out so that when you update your stack to add a new service to the stack, the initial service will get the data it was initiated with. (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.)
Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data.
To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before...
All that combined means I haven't figured out how I can run a Galera cluster on Swarm. Even if I use a service per instance, updates are going to fail unless I do some deep customization on the Galera image I'm using to make it use unique data directories per startup. Even if I succeed in that, I'll still have to figure out how to clean out old data... I mean, I could manually add a new volume and service, then manually remove the old volume and service for each instance of Galera I'm running. But at the point, why bother with containers?
Anyway, I'm pretty sure I've done my research and am correct on all of this, but I'd be happy to be proven wrong. Swarm/Portainer/Traefik is a really really nice stack...
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
