Nomad VS k3s

[Nomad]

Lob’s core API has been fully migrated to HashiCorp's Nomad, Lob’s Next Generation service platform. This is a major milestone for the Nomad Project, the Platform Team, and Lob Engineering. This migration is the culmination of a year of R&D, months of practice migrating other Lob services, and weeks of work on this particular service. It’s absolutely worth celebrating for the complexity and customer impact.

Most of our API’s migration to Nomad happened in August, but the key feature of autoscaling was not working as expected. This turned out to be a bug in Nomad which James Douglas tracked down. The issue was recently fixed and autoscaling works as expected, completing the migration!

Nomad by HashiCorp

There's also https://pot.pizzamig.dev/ which even has a Nomad(https://www.nomadproject.io/) driver in case you want to "orchestrate" the jails(as in deploy many jails at the same time across multiple servers)

The check stanza runs a health check on the service Since the service is registered to Consul, the health check runs on Consul. The above health check is configured to run every 10 seconds, and is given 5 seconds for the health check query to succeed. Health checks in Nomad are similar to Kubernetes health checks.

Not sure if it fits here, and might be a little more complex than what you’re looking for, but Nomad is definitely worth a look.

I'd like to also mention hashicorp nomad + consul. They're worth evaluating.

Hashicorp Nomad

We had a hard time finding this data when experimenting with carbon aware scheduling in Nomad. There seem to be a lot of orgs working on this, but it’s all third party as far as I know. I couldn’t find a first party (eg Amazon, Microsoft, Google, etc) that provided this data via an API. Very frustrating.

There are a couple resources linked here: https://github.com/hashicorp/nomad/blob/h-carbon-meta/CARBON...

The silver lining is that the folks who are working in this space are extremely friendly and passionate in my experience. Don’t be scared to “contact sales for an API let.” The few folks I talked to were extremely helpful.

Then along came Scaleway with their very cheap ARM clouds and K3S with a lightweight Kubernetes that was perfect for Raspberry Pis. Now you can have very cost-effective and (with enough nodes) high-performing clusters running on machines lying around your office. These are perfect for development and staging clusters to test out your applications.

I'd say that going from Docker Compose to Docker Swarm is the first logical step, because it's included in a Docker install and also uses the same Compose format (with more parameters, such as deployment constraints, like which node hostname or tag you want a certain container to be scheduled on): https://docs.docker.com/compose/compose-file/compose-file-v3... That said, you won't see lots of Docker Swarm professionally anymore - it's just the way the job market is, despite it being completely sufficient for many smaller projects out there, I'm running it in prod successfully so far and it's great.

Another reasonably lightweight alternative would be Hashicorp Nomad, because it's free, simple to deploy and their HCL format isn't too bad either, as long as you keep things simple, in addition to them supporting more than just container workloads: https://www.hashicorp.com/products/nomad That said, if you don't buy into HashiStack too much, then there won't be too much benefit from learning HCL and translating the contents of various example docker-compose.yml files that you see in a variety of repos out there, although their other tools are nice - for example, Consul (a service mesh). This is a nice but also a bit niche option.

Lastly, there is Kubernetes. It's complicated, even more so when you get into solutions like Istio, typically eats up lots of resources, can be difficult to manage and debug, but does pretty much anything that you might need, as long as you have either enough people to administer it, or a wallet that's thick enough for you to pay one of the cloud vendors to do it for you. Personally, I'd look into the lightweight clusters at first, like k0s, MicroK8s, or perhaps the K3s project in particular: https://k3s.io/

I'd also suggest that if you get this far, don't be afraid to look into options for dashboards and web based UIs to make exploring things easier:

  - for Docker Swarm and Kubernetes there is Portainer: https://www.portainer.io/

Here is the best part! K3d is a wrapper around k3s and can set up your entire cluster using Docker in no time. It should be noted that this is not intended for production, but it is intended for tinkerers, learning, and exam prep. Here's an example command and then we will break it down:

k3s

> I tried portainer, awful UX experience and all good features are inside paid version.

This is interesting to me, because it doesn't quite match my experience - I've been using Portainer for around 3 years at this point and it's been pretty decent.

The worst issues that I've gotten is networking issues in some hybrid configurations with Docker Swarm (e.g. Portainer cannot reach the manager node of the cluster for a bit), or troubles configuring Traefik ingresses when managing Kubernetes (though I think the recent patch notes talked about improving the ingress section, so maybe the experience will get better with non-Nginx ingresses).

Other than that, it's been great for onboarding new people, illustrating the cluster state at a glance, easily operating with stacks and scaling/restarting services as needed, including pulling new images, viewing the logs or even connecting to containers through a web UI if need be. The webhook functionality in particular is really nice - you can just do a curl request against a given URL and that will pull the new container versions for the given image and do a redeploy, which works nicely with a variety of CI solutions.

When I last tried, initializing Nomad clusters with networking encryption was a bit less of a smooth experience (needing to essentially manage your own PKI) and the web UI felt more like a dashboard, instead of something that you could click around in, if you're a proponent of that workflow.

Rancher is probably better than both of those options, though there's a certain overhead in regards to running both that software and a full Kubernetes cluster. If Kubernetes feels like a good fit for a particular project and resources aren't an issue, definitely check it out! You can, of course, also have some success with lightweight clusters, like K3s: https://k3s.io/

I'll definitely agree that Lazydocker is a nice tool, but I wouldn't call it superior, just different (TUI vs GUI), their demo video is nice though: https://youtu.be/NICqQPxwJWw

It actually reminds me of ctop, which you might also want to check out, though it's not something that you'd manage clusters in, merely the individual containers on a node (which won't always be enough, same as Docker Compose isn't): https://github.com/bcicen/ctop

Regardless, for Kubernetes, I'm inclined to say that you'd enjoy k9s a bunch then, it has a similar TUI approach: https://k9scli.io/

$ k3sup install --ip $IP --user $USER Running: k3sup install 2023/01/10 01:10:17 44.204.142.40 Public IP: 192.168.0.1 [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service. [INFO] systemd: Starting k3s Result: [INFO] Finding release for channel stable [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit [INFO] systemd: Starting k3s Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service. Saving file to: /home/ec2-user/kubeconfig # Test your cluster with: export KUBECONFIG=/home/roxsross/kubeconfig kubectl config use-context default kubectl get node -o wide 🐳 k3sup needs your support: https://github.com/sponsors/alexellisll

I go with k8s even on a single server nowadays, it just makes everything so much more convenient.

https://k3s.io/ makes it really easy to set up, too.

Stand-alone k3s, https://k3s.io