Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
external-dns
Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
-
-
awesome-home-kubernetes
Discontinued ⚠️ Deprecated: Awesome projects involving running Kubernetes at home
Been running my blog on Kubernetes since 2017[0] and it's been great. Kubernetes is a force multiplier and the natural evolution in deployment methodology. It could be tighter/cleaner but it's extremely good for what it provides, but you have to know why you might need the things it's providing for it to seem worth it.
If you'd like to try kubernetes and get to understand it and feel comfortable with it I'd recommend:
- Work through kubernetes the hard way[1] (ignore/replace the GCP-specific things, if you don't know enough about what the non-GCP corrolary to a GCP-thing is, that's a bit of knowledge you need to fill)
- Set up your own cluster from scratch (no kubeadm, no alternate distros, etc), use the simplest options you can find at first (ex. flannel for CNI)
- Set up ingress (NGINX ingress is a good place to start)
- Run some simple unsecured but diverse workloads (static sites on NGINX, Wordpress, etc), figure out why you might pick a StatefulSet versus DaemonSet. At this point, use the hostPath/local volumes just to avoid trying to grok volume complexity.
- Install a useful cluster tool ("addon") like cert-manager[2] from scratch so you can see Kubernetes manage something you'd normally solve with systemd timers and `certbot` (or your reverse proxy would do for you if you're running caddy or traefik)
- Start putting your YAML in source control get familiar with either kustomize, helm, or both (you could also just use Make + envsubst like I did for a while[3]). It's at this point that it should click that all you need to get back to a certain state of your cluster is to get a machine, do basic hardening (ufw, etc), install kubernetes, and run "make" in this repo (excluding things like DNS entries, etc). Now things are probably getting fun, because you can have the distant cousin of immutable infrastructure; repeatable infrastructure.
- Tear down your cluster, set it up again with kubeadm (note that kubeadm actually has a file-driven configuration option[3]), run your yaml from source control and confirm that all the workloads you had in place are back up and secured.
- (optional) Tear down your cluster, try rebuilding it with k3s[4] or k0s[5]
- Start looking around and seeing what your options are and the ecosystem that exists -- digging deeper into the interfaces that make Kubernetes tick, for example volume management (Container Storage Interface) by deploying Rook[6] or OpenEBS[7].
I've made a guide like this before, I'll see if I can find it.
[0]: https://vadosware.io/post/fresh-dedicated-server-to-single-n...
[1]: https://github.com/kelseyhightower/kubernetes-the-hard-way
[2]: https://github.com/jetstack/cert-manager
[3]: https://www.vadosware.io/post/using-makefiles-and-envsubst-a...
[4]: https://k3s.io/
[5]: https://docs.k0sproject.io
[6]: https://rook.io/docs
[7]: https://docs.openebs.io/
These days I see no point in installing services on a VPS directly, other than docker + docker-compose. You could do it in one image with the reverse proxy + static files, or break it out into two images (this is helpful if you run more services on the VPS).
As for updates, caddy v1 used to support pulling in from git, but I don't think that got ported to v2. So what I do is build+push a docker image, and have a cron job on my vps to pull+restart my website.
My preferences go Traefik > Caddy > Nginx, but traefik definitely has a bit of a learning curve.
https://github.com/andrewzah/andrewzah-com-source/blob/maste...
https://github.com/andrewzah/andrewzah.com-docker/tree/maste...
I've had a great experience with https://rook.io/ managing Ceph for Kubernetes in production.
Next step: unironically creating a Kubernetes Operator for your personal blog https://github.com/dexhorthy/captains-log
I run GKE for some small apps. I also use AWS S3 hosting for my personal blog. The cost differences are... non trivial to the point of a bad joke, if we were comparing ability to reliably ship plaintext over the wire. But I'm not. I host a database and webapps on the k8s cluster, without adding extra EC2 nodes, RDS costs, or wrestling with AWS Lambda limitations.
I can also confidently say that having something approximating a stable web app demands doing a lot of serious thinking, and "a single server running Apache on Digital Ocean" does not cover that case sufficiently. You need to tolerate failure, failover, load balancing, bin-packing, etc. I used to run a small autoscaling group on EC2 for my own systems; the dang thing would fail to come up on one node very frequently and so a number of the queries would fail. I eventually burnt it to the ground and redid it. I've never had that hassle in k8s. Its designed to succeed, in a way the "box of parts" approach doesn't.
Boxes of parts are useful. For a complexity-sensitive & thoughtful infrastructure engineer, having something like the old Synapse/Nerve[1] system with your apps distributed across some 5-20 machines with a monitor lease to spawn new ones on failure would probably approximate Kubernetes for a few years, until you have to do something fancypants. You've still reimplemented part of Kubernetes, though... The other angle is, boxes of parts can go in wildly weird directions.... if you need it.
Looking at some infrastructure these days professionally, the question is - when do we move to Kubernetes. It's not interesting or useful to the company to be maintaining our own thing or own strange path. The only questions are around the path - how much rework needs to happen and how much building in k8s needs to happen to get there.
GKE is a very good starting point for k8s. Strong recommend.
https://github.com/airbnb/nerve
You might be interested in this project that I found in another HN comment recently:
https://github.com/containrrr/watchtower
It would handle automatically restarting your containers when you push a new image.
YAML is the biggest flaw of Kubernetes, so I'm quite exited that cdk8s is progressing very nicely https://github.com/cdk8s-team/cdk8s
There are other solutions that are potentially better (e.g. Dhall) but cdk8s seems to have momentum and sense for tackling the practical stuff (integrates easily with cdk, library with simplified constructs cdk8s-plus, import and convert existing stuff easily etc)
> Is there a service for autoconfiguring the DNS
There is! I use external-dns. [1]
I haven't actually set up a Let's Encrypt wildcard cert, but I'm pretty certain cert-manager [2] supports it. I don't think you need a proxy if you use the DNS01 challenges.
[1] https://github.com/kubernetes-sigs/external-dns/
[2] https://cert-manager.io/docs/
I had bad experience with k8s. The learning curve is step. A few days ago k8s decided to run a cron job just after the deploy (it was scheduled for Sunday). Why? it could be a bug (https://github.com/kubernetes/kubernetes/issues/63371) I'm not sure how to even debug this.
> A cron job creates a job object about once per execution time of its schedule. We say "about" because there are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare, but do not completely prevent them. Therefore, jobs should be idempotent
Is there a way to make a cron job idempotent?
> There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc.
This is an unbelievable amount of gate-keeping hogwash. I don't know who this person thinks they can arbitrate what is a good usage & what cases this is too-powerful too-interesting too-useful to bother using it in.
There is so so so much fear & doubt & scare in this post. Screw this gate-keeping crap.
> Maybe, if you're lucky, you had some set of really good and reliable Sys Admins that figured out a robust way to script and configure the setup process of your original on-premise data center and they captured that in very good, well-maintained documentation. If you're even luckier, maybe those same guys still work for you.
"Only us good right & virtuous & amazing engineers can handle this! This is too pure, too amazing for mortals! They're wasting their time! They'll get the configuration wrong! They're bad people. Only professionals are qualified to play with Kubernetes!"
UGH ENOUGH. Stop this terrible attitude. This is so down-talk-y.
Please don't assume, please don't dictate your limited terms to the world. Let the world try. Let us not be cowed, & afraid, to use good tech, by these scare words.
As it turns out, it's just not that hard. It's a better environment, a better world. There are lots of home users using Kubernetes. It doesn't take a colossal investment. It's fairly secure out of the box, at least if you're not trying to run a multi-tenant home.
See? Look. Lots of projects: https://github.com/k8s-at-home/awesome-home-kubernetes . Lots. Good people, just trying. Not taking the poison words to be afraid, that this is too hard.
"For one person, this means nothing. This guy is just doing it for fun. "
What a BAD ATTITUDE. Snarking & being mean, to people out there, trying to find better ways to do things, to create shared, meaningful value. With good, autonomic systems. With reasonably competent free-to-everyone utility scale / cloud computing. Don't accept such words as these. Do not be afraid to involve yourselves. Do not be gate-kept like this. Run Kubernetes. Run good systems. Stop being sold on second, third run systems. You are not saving yourselves any hardship. You can run K3S in <20 minutes, and you can start loading amazing Charts seconds after. Please allow yourself a moment of un-doubt where you consider, maybe, this has amazing value for the home, that it's already possibly incredibly robust, please consider that applying some manifests might be super easy. Please consider that blog posts are the canonical way to share work, before Kubernetes, but now I can link you to a repo full of people sharing manifests & charts & works, that stand a decent chance of running on any cloud or at home. There's a lot of sophisticated under-the-hood boons to running Kubernetes too, but as for what the home-user gains: it's amazing. And growing.