Nomad VS Portainer

[Nomad]

Lob’s core API has been fully migrated to HashiCorp's Nomad, Lob’s Next Generation service platform. This is a major milestone for the Nomad Project, the Platform Team, and Lob Engineering. This migration is the culmination of a year of R&D, months of practice migrating other Lob services, and weeks of work on this particular service. It’s absolutely worth celebrating for the complexity and customer impact.

Most of our API’s migration to Nomad happened in August, but the key feature of autoscaling was not working as expected. This turned out to be a bug in Nomad which James Douglas tracked down. The issue was recently fixed and autoscaling works as expected, completing the migration!

Nomad by HashiCorp

There's also https://pot.pizzamig.dev/ which even has a Nomad(https://www.nomadproject.io/) driver in case you want to "orchestrate" the jails(as in deploy many jails at the same time across multiple servers)

The check stanza runs a health check on the service Since the service is registered to Consul, the health check runs on Consul. The above health check is configured to run every 10 seconds, and is given 5 seconds for the health check query to succeed. Health checks in Nomad are similar to Kubernetes health checks.

Not sure if it fits here, and might be a little more complex than what you’re looking for, but Nomad is definitely worth a look.

I'd like to also mention hashicorp nomad + consul. They're worth evaluating.

Hashicorp Nomad

We had a hard time finding this data when experimenting with carbon aware scheduling in Nomad. There seem to be a lot of orgs working on this, but it’s all third party as far as I know. I couldn’t find a first party (eg Amazon, Microsoft, Google, etc) that provided this data via an API. Very frustrating.

There are a couple resources linked here: https://github.com/hashicorp/nomad/blob/h-carbon-meta/CARBON...

The silver lining is that the folks who are working in this space are extremely friendly and passionate in my experience. Don’t be scared to “contact sales for an API let.” The few folks I talked to were extremely helpful.

I use Hetzner, Contabo, Time4VPS and other platforms in pretty much the same way (as IaaS VPS providers on top of which I run software, as opposed to SaaS/PaaS), but here's a quick glance at how I do things.

> deploy from source repo? Terraform?

Personally, I use Gitea for my repos and Drone CI for CI/CD.

Gitea: https://gitea.io/en-us/

Drone CI: https://www.drone.io/

Some might prefer Woodpecker due to licensing: https://woodpecker-ci.org/ but honestly most solutions out there are okay, even Jenkins.

Then I have some sort of a container cluster on the servers, so I can easily deploy things: I still like Docker Swarm (projects like CapRover might be nice to look at as well), though many might enjoy the likes of K3s or K0s more (lightweight Kubernetes clusters).

Docker Swarm: https://docs.docker.com/engine/swarm/ (uses the Compose spec for manifests)

K3s: https://k3s.io/

K0s: https://k0sproject.io/ though MicroK8s and others are also okay.

I also like having something like Portainer to have a GUI to manage the clusters: https://www.portainer.io/ for Kubernetes Rancher might offer more features, but will have a higher footprint

It even supports webhooks, so I can do a POST request at the end of a CI run and the cluster will automatically pull and launch the latest tagged version of my apps: https://docs.portainer.io/user/docker/services/webhooks

> keep software up to date? ex: Postgres, OS

I build my own base container images and rebuild them (with recent package versions) on a regular basis, which is automatically scheduled: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...

Drone CI makes this easy to have happen in the background, as long as I don't update across major versions, or Maven decides to release a new version and remove their old version .tar.gz archives from the downloads site for some reason, breaking my builds and making me update the URL: https://docs.drone.io/cron/

Some images like databases etc. I just proxy to my Nexus instance, version upgrades are relatively painless most of the time, at least as long as I've set up the persistent data directories correctly.

> do load balancing? built-in load balancer?

This is a bit more tricky. I use Apache2 with mod_md to get Let's Encrypt certificates and Docker Swarm networking for directing the incoming traffic across the services: https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-...

Some might prefer Caddy, which is another great web server with automatic HTTPS: https://caddyserver.com/ but the Apache modules do pretty much everything I need and the performance has never actually been too bad for my needs. Up until now, applications themselves have always been the bottleneck, actually working on a blog post about comparing some web servers in real world circumstances.

However, making things a bit more failure resilient might involve just paying Hetzner (in this case) to give you a load balancer: https://www.hetzner.com/cloud/load-balancer which will make everything less painless once you need to scale.

Why? Because doing round robin DNS with the ACME certificate directory accessible and synchronized across multiple servers is a nuisance, although servers like Caddy attempt to get this working: https://caddyserver.com/docs/automatic-https#storage You could also get DNS-01 challenges working, but that needs even more work and integration with setting up TXT records. Even if you have multiple servers for resiliency, not all clients would try all of the IP addresses if one of the servers is down, although browsers should: https://webmasters.stackexchange.com/a/12704

So if you care about HTTPS certificates and want to do it yourself with multiple servers having the same hostname, you'll either need to get DNS-01 working, do some messing around with shared directories (which may or may not actually work), or will just need to get a regular commercial cert that you'd manually propagate to all of the web servers.

From there on out it should be a regular reverse proxy setup, in my case Docker Swarm takes care of the service discovery (hostnames that I can access).

> handle scaling? Terraform?

None, I manually provision how many nodes I need, mostly because I'm too broke to hand over my wallet to automation.

They have an API that you or someone else could probably hook up: https://docs.hetzner.cloud/

> automate backups? ex: databases, storage. Do you use provided backups and snapshots?

I use bind mounts for all of my containers for persistent storage, so the data is accessible on the host directly.

Then I use something like BackupPC to connect to those servers (SSH/rsync) and pull data to my own backup node, which then compresses and deduplicates the data: https://backuppc.github.io/backuppc/

It was a pain to setup, but it works really well and has saved my hide dozens of times. Some might enjoy Bacula more: https://www.bacula.org/

> maintain security? built-in firewall and DDoS protection?

I personally use Apache2 with ModSecurity and the OWASP ruleset, to act as a lightweight WAF: https://owasp.org/www-project-modsecurity-core-rule-set/

You might want to just cave in and go with Cloudflare for the most part, though: https://www.cloudflare.com/waf/

There’s an option to make that yellow upgrade button more subtle in 2.17.1. https://github.com/portainer/portainer/issues/8517

if you need a visual way to understand this, try portainer (https://www.portainer.io/). the community edition is free

FWIW, upgrading my Synology installation manually "worked," but then I ran into known issue #8461 and had to roll back to CE. Unfortunately, when attempting to roll back the database as described here, it resulted in "failed rolling back to CE | error='object not found inside the database'". Replaced the DB with a backup, fired up the CE container, and all is well, albeit with a fair amount of time lost in troubleshooting and rolling back. Hope this gets sorted out soon.

We're aware of some issues with the in-app upgrade on Synology platforms at the moment and have an internal issue open around this. We're tracking this in our Github issues and in the meantime you can do a manual upgrade - have a look at our documentation for info on this.

If an app can be installed through the Store, I'll do it in the Store. Only if it's unvailable that way, will I use the terminal. An example I'd use the terminal with, would be OpenSnitch. I also use the GUI for docker, with Portainer as a frontend.

There's an option to make this more subtle in 2.17.1. https://github.com/portainer/portainer/issues/8517

Without being too cheeky, it certainly isn't the new ads that are implemented for BE upselling: https://github.com/portainer/portainer/issues/8452