vault-secrets-operator VS ignite

For home use, I wouldn't bother with Vault unless that's really what you want to learn. Then it's worth looking into setting something up where you could use vault secrets, using one of the available options (I haven't seen the vault-secrets-operator being mentioned).

It is but it affects vault-secrets-operator too, see https://github.com/ricoberger/vault-secrets-operator/issues/104 (and no, I’ve only use vault-secrets-operator)

What you’re looking for is https://github.com/ricoberger/vault-secrets-operator

If you are using an external KMS in any case, then there are other options, such as the kubernetes-external-secrets operator that was originally started by GoDaddy and the externalsecret-operator from Container Solutions. If you use HashiCorp Vault, you also have the option of using the Vault Secrets operator. This works similarly to the Sealed Secrets Operator, but instead of managing its own key material, it retrieves the secrets from Vault. The CNCF Technology Radar from January 2021 provides an overview of the types of tools that are available for secrets management.

https://github.com/weaveworks/ignite ?

Not a drop-in replacement: the OCI image entry point is not automatically executed. https://github.com/weaveworks/ignite/issues/874 (issue opened in 2021).

Resource usage. Running a full VM to execute only two processes is a big waste of the limited resources of my server (8GB RAM, Celeron processor). I searched and I found something to limit the resource usage and minimize the attack surface of the guest OS: Weave Ignite/Firecracker/MicroVMs.

root@kubefire:~# kubefire install INFO[2022-11-11T11:46:13Z] downloading https://raw.githubusercontent.com/innobead/kubefire/v0.3.8/scripts/install-prerequisites.sh to save /root/.kubefire/bin/v0.3.8/install-prerequisites.sh force=false version=v0.3.8 INFO[2022-11-11T11:46:14Z] running script (install-prerequisites.sh) version=v0.3.8 INFO[2022-11-11T11:46:14Z] running /root/.kubefire/bin/v0.3.8/install-prerequisites.sh version=v0.3.8 INFO[2022-11-11T11:46:14Z] + TMP_DIR=/tmp/kubefire INFO[2022-11-11T11:46:14Z] ++ go env GOARCH INFO[2022-11-11T11:46:14Z] ++ echo amd64 INFO[2022-11-11T11:46:14Z] + GOARCH=amd64 INFO[2022-11-11T11:46:14Z] + KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + CONTAINERD_VERSION=v1.6.6 + IGNITE_VERION=v0.10.0 INFO[2022-11-11T11:46:14Z] + CNI_VERSION=v1.1.1 + RUNC_VERSION=v1.1.3 INFO[2022-11-11T11:46:14Z] + '[' -z v0.3.8 ']' + '[' -z v1.6.6 ']' + '[' -z v0.10.0 ']' + '[' -z v1.1.1 ']' + '[' -z v1.1.3 ']' INFO[2022-11-11T11:46:14Z] ++ sed -E 's/(v[0-9]+\.[0-9]+\.[0-9]+)[a-zA-Z0-9\-]*/\1/g' INFO[2022-11-11T11:46:14Z] +++ echo v0.3.8 INFO[2022-11-11T11:46:14Z] + STABLE_KUBEFIRE_VERSION=v0.3.8 INFO[2022-11-11T11:46:14Z] + rm -rf /tmp/kubefire INFO[2022-11-11T11:46:14Z] + mkdir -p /tmp/kubefire INFO[2022-11-11T11:46:14Z] + pushd /tmp/kubefire /tmp/kubefire /root INFO[2022-11-11T11:46:14Z] + trap cleanup EXIT ERR INT TERM INFO[2022-11-11T11:46:14Z] + check_virtualization + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + lscpu INFO[2022-11-11T11:46:14Z] + grep 'Virtuali[s|z]ation' INFO[2022-11-11T11:46:14Z] Virtualization: VT-x Virtualization type: full INFO[2022-11-11T11:46:14Z] + lsmod INFO[2022-11-11T11:46:14Z] + grep kvm INFO[2022-11-11T11:46:14Z] kvm_intel 372736 0 kvm 1028096 1 kvm_intel INFO[2022-11-11T11:46:14Z] + install_runc + _check_version /usr/local/bin/runc -version v1.1.3 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/runc + local exec_version_cmd=-version + local version=v1.1.3 + command -v /usr/local/bin/runc + return 1 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSL https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64 -o runc INFO[2022-11-11T11:46:14Z] + chmod +x runc INFO[2022-11-11T11:46:14Z] + sudo mv runc /usr/local/bin/ INFO[2022-11-11T11:46:14Z] + install_containerd + _check_version /usr/local/bin/containerd --version v1.6.6 INFO[2022-11-11T11:46:14Z] + set +o pipefail + local exec_name=/usr/local/bin/containerd + local exec_version_cmd=--version + local version=v1.6.6 + command -v /usr/local/bin/containerd + return 1 + local version=1.6.6 + local dir=containerd-1.6.6 + _is_arm_arch INFO[2022-11-11T11:46:14Z] + uname -m INFO[2022-11-11T11:46:14Z] + grep aarch64 INFO[2022-11-11T11:46:14Z] + return 1 INFO[2022-11-11T11:46:14Z] + curl -sfSLO https://github.com/containerd/containerd/releases/download/v1.6.6/containerd-1.6.6-linux-amd64.tar.gz INFO[2022-11-11T11:46:15Z] + mkdir -p containerd-1.6.6 INFO[2022-11-11T11:46:15Z] + tar -zxvf containerd-1.6.6-linux-amd64.tar.gz -C containerd-1.6.6 INFO[2022-11-11T11:46:15Z] bin/ bin/containerd-shim INFO[2022-11-11T11:46:15Z] bin/containerd INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v1 INFO[2022-11-11T11:46:16Z] bin/containerd-stress INFO[2022-11-11T11:46:16Z] bin/containerd-shim-runc-v2 INFO[2022-11-11T11:46:16Z] bin/ctr INFO[2022-11-11T11:46:17Z] + chmod +x containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr INFO[2022-11-11T11:46:17Z] + sudo mv containerd-1.6.6/bin/containerd containerd-1.6.6/bin/containerd-shim containerd-1.6.6/bin/containerd-shim-runc-v1 containerd-1.6.6/bin/containerd-shim-runc-v2 containerd-1.6.6/bin/containerd-stress containerd-1.6.6/bin/ctr /usr/local/bin/ INFO[2022-11-11T11:46:17Z] + curl -sfSLO https://raw.githubusercontent.com/containerd/containerd/v1.6.6/containerd.service INFO[2022-11-11T11:46:17Z] + sudo groupadd containerd INFO[2022-11-11T11:46:17Z] + sudo mv containerd.service /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] ++ command -v chgrp INFO[2022-11-11T11:46:17Z] ++ tr -d '\n' INFO[2022-11-11T11:46:17Z] + chgrp_path=/usr/bin/chgrp INFO[2022-11-11T11:46:17Z] + sudo sed -i -E 's#(ExecStart=/usr/local/bin/containerd)#\1\nExecStartPost=/usr/bin/chgrp containerd /run/containerd/containerd.sock#g' /etc/systemd/system/containerd.service INFO[2022-11-11T11:46:17Z] + sudo mkdir -p /etc/containerd INFO[2022-11-11T11:46:17Z] + containerd config default INFO[2022-11-11T11:46:17Z] + sudo tee /etc/containerd/config.toml INFO[2022-11-11T11:46:17Z] + sudo systemctl enable --now containerd INFO[2022-11-11T11:46:17Z] Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. INFO[2022-11-11T11:46:17Z] + install_cni + _check_version /opt/cni/bin/bridge --version v1.1.1 + set +o pipefail INFO[2022-11-11T11:46:17Z] + local exec_name=/opt/cni/bin/bridge + local exec_version_cmd=--version + local version=v1.1.1 + command -v /opt/cni/bin/bridge INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + mkdir -p /opt/cni/bin INFO[2022-11-11T11:46:17Z] + local f=https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz + _is_arm_arch INFO[2022-11-11T11:46:17Z] + uname -m INFO[2022-11-11T11:46:17Z] + grep aarch64 INFO[2022-11-11T11:46:17Z] + return 1 INFO[2022-11-11T11:46:17Z] + curl -sfSL https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz INFO[2022-11-11T11:46:17Z] + tar -C /opt/cni/bin -xz INFO[2022-11-11T11:46:19Z] + install_cni_patches + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -o host-local-rev -sfSL https://github.com/innobead/kubefire/releases/download/v0.3.8/host-local-rev-linux-amd64 INFO[2022-11-11T11:46:19Z] + chmod +x host-local-rev INFO[2022-11-11T11:46:19Z] + sudo mv host-local-rev /opt/cni/bin/ INFO[2022-11-11T11:46:19Z] + install_ignite + _check_version /usr/local/bin/ignite version v0.10.0 + set +o pipefail INFO[2022-11-11T11:46:19Z] + local exec_name=/usr/local/bin/ignite + local exec_version_cmd=version + local version=v0.10.0 + command -v /usr/local/bin/ignite + return 1 INFO[2022-11-11T11:46:19Z] + for binary in ignite ignited + echo 'Installing ignite...' INFO[2022-11-11T11:46:19Z] Installing ignite... INFO[2022-11-11T11:46:19Z] + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 + _is_arm_arch INFO[2022-11-11T11:46:19Z] + uname -m INFO[2022-11-11T11:46:19Z] + grep aarch64 INFO[2022-11-11T11:46:19Z] + return 1 + curl -sfSLo ignite https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignite-amd64 INFO[2022-11-11T11:46:20Z] + chmod +x ignite INFO[2022-11-11T11:46:20Z] + sudo mv ignite /usr/local/bin INFO[2022-11-11T11:46:20Z] + for binary in ignite ignited + echo 'Installing ignited...' Installing ignited... + local f=https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:20Z] + _is_arm_arch INFO[2022-11-11T11:46:20Z] + grep aarch64 + uname -m INFO[2022-11-11T11:46:20Z] + return 1 + curl -sfSLo ignited https://github.com/weaveworks/ignite/releases/download/v0.10.0/ignited-amd64 INFO[2022-11-11T11:46:21Z] + chmod +x ignited INFO[2022-11-11T11:46:21Z] + sudo mv ignited /usr/local/bin INFO[2022-11-11T11:46:21Z] + check_ignite + ignite version INFO[2022-11-11T11:46:21Z] Ignite version: version.Info{Major:"0", Minor:"10", GitVersion:"v0.10.0", GitCommit:"4540abeb9ba6daba32a72ef2b799095c71ebacb0", GitTreeState:"clean", BuildDate:"2021-07-19T20:52:59Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64", SandboxImage:version.Image{Name:"weaveworks/ignite", Tag:"v0.10.0", Delimeter:":"}, KernelImage:version.Image{Name:"weaveworks/ignite-kernel", Tag:"5.10.51", Delimeter:":"}} INFO[2022-11-11T11:46:21Z] Firecracker version: v0.22.4 INFO[2022-11-11T11:46:21Z] + create_cni_default_config INFO[2022-11-11T11:46:21Z] + mkdir -p /etc/cni/net.d/ INFO[2022-11-11T11:46:21Z] + sudo cat INFO[2022-11-11T11:46:21Z] + popd /root + cleanup INFO[2022-11-11T11:46:21Z] + rm -rf /tmp/kubefire

> If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one?

I already did that with live migration. But ok.

Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX

The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things.

The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works.

VLAN: https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day.

Again, trivial for a full-featured Linux kernel.

I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.

This is a great idea and one I would love to use, but it's a lot less docker compatible than you may think. For instance there is no way to automatically run the specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874

> So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close).

This is pretty much me and my homelab. I haven't visited it in a while, but Weave Ignite might be of interest here. https://github.com/weaveworks/ignite