harbormaster VS ignite

I use an HP ProLiant Microserver with four drives in a ZFS RAIDZ array and an SSD for the OS. For software, I mostly run it in Docker using a very small container orchestration program I wrote:

https://gitlab.com/stavros/harbormaster

I needed something that would restart containers automatically when I pushed to a branch, so I wrote a few lines of code to do it:

https://gitlab.com/stavros/harbormaster

As far as PaaSes go, it's probably the simplest, and works really well.

I was in the same boat as you and built something simple that I really like:

https://gitlab.com/stavros/harbormaster

It'll just pull some repos, make sure the containers are up, and make your configuration simple and discoverable. It really works great at that.

I do this for our services, it works great and we can easily put SSO in front of them with CF Access. I publish a Docker container that you can use as a sidecar for your Compose deployments:

https://gitlab.com/stavros/docker-cloudflared

I use this with Harbormaster (https://gitlab.com/stavros/harbormaster) so I can expose containerized stuff without ever forwarding any ports outside of Docker.

I use Dokku for that (I can share my Bitwarden repo if you want, the entire thing is four lines or something). I also made https://gitlab.com/stavros/harbormaster for things that weren't so "web server -> app -> database" and love it.

I had the same problem and didn't want to manage things by hand, so I wrote Harbormaster:

https://gitlab.com/stavros/harbormaster

It basically pulls Compose apps from the git repositories you specify, builds the containers and makes sure they're running. Pretty simple and works really well for me.

(This post should read "Argo tunnel" instead of just "Argo")

I did the same to enable secure access to services via SSO at work. I used Harbormaster[1] to deploy Compose files, but it's otherwise the same setup.

One of the big advantages this has is that the services can't be accessed any other way (not even from the same host, as they only listen inside the Docker network). That makes it hard to forget some port exposed because you listened to 0.0.0.0 instead of localhost.

Cloudflare access is very easy to set up SSO with, as well. I'd recommend this setup if you need it, though for home usage I usually just set up Caddy as a reverse proxy with basic auth, as I'll be the only person using this and I don't want Cloudflare MITMing my personal stuff.

[1]: https://gitlab.com/stavros/harbormaster

Something like harbormaster? https://gitlab.com/stavros/harbormaster

Not a drop-in replacement: the OCI image entry point is not automatically executed. https://github.com/weaveworks/ignite/issues/874 (issue opened in 2021).

Resource usage. Running a full VM to execute only two processes is a big waste of the limited resources of my server (8GB RAM, Celeron processor). I searched and I found something to limit the resource usage and minimize the attack surface of the guest OS: Weave Ignite/Firecracker/MicroVMs.

root@kubefire:~# kubefire install INFO[2022-11-11T11:46:13Z] downloading https://raw.githubusercontent.com/innobead/kubefire/v0.3.8/scripts/install-prerequisites.sh to save /root/.kubefire/bin/v0.3.8/install-prerequisites.sh force=false version=v0.3.8 INFO[2022-11-11T11:46:14Z] running script (install-prerequisites.sh) version=v0.3.8 INFO[2022-11-11T11:46:14Z] running /root/.kubefire/bin/v0.3.8/install-prerequisites.sh version=v0.3.8 INFO[2022-11-11T11:46:14Z] + TMP_DIR=/tmp/kubefire INFO[2022-11-11T11:46:14Z] ++ go env GOARCH INFO[2022-11-11T11:46:14Z] ++ echo amd64 INFO[2022-11-11T11:46:14Z] + GOARCH=amd64 INFO[2022-11-11T11:46:14Z] + KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + CONTAINERD_VERSION=v1.6.6 + IGNITE_VERION=v0.10.0 INFO[2022-11-11T11:46:14Z] + CNI_VERSION=v1.1.1 + RUNC_VERSION=v1.1.3 INFO[2022-11-11T11:46:14Z] + '[' -z v0.3.8 ']' + '[' -z v1.6.6 ']' + '[' -z v0.10.0 ']' + '[' -z v1.1.1 ']' + '[' -z v1.1.3 ']' INFO[2022-11-11T11:46:14Z] ++ sed -E 's/(v[0-9]+\.[0-9]+\.[0-9]+)[a-zA-Z0-9\-]*/\1/g' INFO[2022-11-11T11:46:14Z] +++ echo v0.3.8 INFO[2022-11-11T11:46:14Z] + STABLE_KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + rm -rf /tmp/kubefire INFO[2022-11-11T11:46:14Z] + mkdir -p /tmp/kubefire INFO[2022-11-11T11:46:14Z] + pushd /tmp/kubefire /tmp/kubefire /root INFO[2022-11-11T11:46:14Z] + trap cleanup EXIT ERR INT TERM INFO[2022-11-11T11:46:14Z] + check_virtualization + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + lscpu INFO[2022-11-11T11:46:14Z] + grep 'Virtuali[s|z]ation' INFO[2022-11-11T11:46:14Z] Virtualization: VT-x Virtualization type: full INFO[2022-11-11T11:46:14Z] + lsmod INFO[2022-11-11T11:46:14Z] + grep kvm INFO[2022-11-11T11:46:14Z] kvm_intel 372736 0 kvm 1028096 1 kvm_intel INFO[2022-11-11T11:46:14Z] + install_runc + _check_version /usr/local/bin/runc -version v1.1.3 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/runc + local exec_version_cmd=-version + local version=v1.1.3 + command -v /usr/local/bin/runc + return 1 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSL https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64 -o runc INFO[2022-11-11T11:46:14Z] + chmod +x runc INFO[2022-11-11T11:46:14Z] + sudo mv runc /usr/local/bin/ INFO[2022-11-11T11:46:14Z] + install_containerd + _check_version /usr/local/bin/containerd --version v1.6.6 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/containerd + local exec_version_cmd=--version + local version=v1.6.6 + command -v /usr/local/bin/containerd + return 1 + local version=1.6.6 + local dir=containerd-1.6.6 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSLO https://github.com/containerd/containerd/releases/download/v1.6.6/containerd-1.6.6-linux-amd64.tar.gz INFO[2022-11-11T11:46:15Z] + mkdir -p containerd-1.6.6 INFO[2022-11-11T11:46:15Z] + tar -zxvf containerd-1.6.6-linux-amd64.tar.gz -C containerd-1.6.6 INFO[2022-11-11T11:46:15Z] bin/ bin/containerd-shim INFO[2022-11-11T11:46:15Z] bin/containerd INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v1 INFO[2022-11-11T11:46:16Z] bin/containerd-stress INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v2 INFO[2022-11-11T11:46:16Z] bin/ctr INFO[2022-11-11T11:46:17Z] + chmod +x containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr INFO[2022-11-11T11:46:17Z] + sudo mv containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr /usr/local/bin/ INFO[2022-11-11T11:46:17Z] + curl -sfSLO https://raw.githubusercontent.com/containerd/containerd/v1.6.6/containerd.service INFO[2022-11-11T11:46:17Z] + sudo groupadd containerd INFO[2022-11-11T11:46:17Z] + sudo mv containerd.service /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] ++ command -v chgrp INFO[2022-11-11T11:46:17Z] ++ tr -d '\n' INFO[2022-11-11T11:46:17Z] + chgrp_path=/usr/bin/chgrp INFO[2022-11-11T11:46:17Z] + sudo sed -i -E 's#(ExecStart=/usr/local/bin/containerd)#\1\nExecStartPost=/usr/bin/chgrp containerd /run/containerd/containerd.sock#g' /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] + sudo mkdir -p /etc/containerd INFO[2022-11-11T11:46:17Z] + containerd config default INFO[2022-11-11T11:46:17Z] + sudo tee /etc/containerd/config.toml INFO[2022-11-11T11:46:17Z] + sudo systemctl enable --now containerd INFO[2022-11-11T11:46:17Z] Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. INFO[2022-11-11T11:46:17Z] + install_cni + _check_version /opt/cni/bin/bridge --version v1.1.1 + set +o pipefail INFO[2022-11-11T11:46:17Z] + local exec_name=/opt/cni/bin/bridge + local exec_version_cmd=--version + local version=v1.1.1 + command -v /opt/cni/bin/bridge INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + mkdir -p /opt/cni/bin INFO[2022-11-11T11:46:17Z] + local f=https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz + _is_arm_arch INFO[2022-11-11T11:46:17Z] + uname -m INFO[2022-11-11T11:46:17Z] + grep aarch64 INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + curl -sfSL https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz INFO[2022-11-11T11:46:17Z] + tar -C /opt/cni/bin -xz INFO[2022-11-11T11:46:19Z] + install_cni_patches + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -o host-local-rev -sfSL https://github.com/innobead/kubefire/releases/download/v0.3.8/host-local-rev-linux-amd64 INFO[2022-11-11T11:46:19Z] + chmod +x host-local-rev INFO[2022-11-11T11:46:19Z] + sudo mv host-local-rev /opt/cni/bin/ INFO[2022-11-11T11:46:19Z] + install_ignite + _check_version /usr/local/bin/ignite version v0.10.0 + set +o pipefail INFO[2022-11-11T11:46:19Z] + local exec_name=/usr/local/bin/ignite + local exec_version_cmd=version + local version=v0.10.0 + command -v /usr/local/bin/ignite + return 1 INFO[2022-11-11T11:46:19Z] + for binary in ignite ignited + echo 'Installing ignite...' INFO[2022-11-11T11:46:19Z] Installing ignite... INFO[2022-11-11T11:46:19Z] + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -sfSLo ignite https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 INFO[2022-11-11T11:46:20Z] + chmod +x ignite INFO[2022-11-11T11:46:20Z] + sudo mv ignite /usr/local/bin INFO[2022-11-11T11:46:20Z] + for binary in ignite ignited + echo 'Installing ignited...' Installing ignited... + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:20Z] + _is_arm_arch INFO[2022-11-11T11:46:20Z] + grep aarch64 + uname -m INFO[2022-11-11T11:46:20Z] + return 1 + curl -sfSLo ignited https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:21Z] + chmod +x ignited INFO[2022-11-11T11:46:21Z] + sudo mv ignited /usr/local/bin INFO[2022-11-11T11:46:21Z] + check_ignite + ignite version INFO[2022-11-11T11:46:21Z] Ignite version: version.Info{Major:"0", Minor:"10", GitVersion:"v0.10.0", GitCommit:"4540abeb9ba6daba32a72ef2b799095c71ebacb0", GitTreeState:"clean", BuildDate:"2021-07-19T20:52:59Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64", SandboxImage:version.Image{Name:"weaveworks/ignite", Tag:"v0.10.0", Delimeter:":"}, KernelImage:version.Image{Name:"weaveworks/ignite-kernel", Tag:"5.10.51", Delimeter:":"}} INFO[2022-11-11T11:46:21Z] Firecracker version: v0.22.4 INFO[2022-11-11T11:46:21Z] + create_cni_default_config INFO[2022-11-11T11:46:21Z] + mkdir -p /etc/cni/net.d/ INFO[2022-11-11T11:46:21Z] + sudo cat INFO[2022-11-11T11:46:21Z] + popd /root + cleanup INFO[2022-11-11T11:46:21Z] + rm -rf /tmp/kubefire

> If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one?

I already did that with live migration. But ok.

Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX

The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things.

The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works.

VLAN: https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day.

Again, trivial for a full-featured Linux kernel.

I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.