qemu-user-static VS k3s

and its "oh, you want multi-arch, do you?" friend. While prosecuting this <https://gitlab.com/gitlab-org/gitlab/-/issues/339567> I learned that https://hub.docker.com/layers/multiarch/qemu-user-static/7.2... actually mutates the binfmt_misc in buildx's context in order to exec the static copy of qemu in it https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...

and, that the buildx plugin itself has some qemu magick in it, which got addressed in a minor version bump but I couldn't track down the relevant GitHub issue this second (I've flushed it from my mind, only recalling that there were a lot of actors in that tire fire)

deploy: <<: *build stage: deploy image: docker:20-cli script: - touch .env - export LOCAL_IMAGE="${CI_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_TAG}" - wget https://github.com/multiarch/qemu-user-static/releases/download/v7.0.0-7/x86_64_qemu-aarch64-static.tar.gz - tar xzf x86_64_qemu-aarch64-static.tar.gz -C /usr/bin/ - docker buildx create --platform linux/arm64 --use - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} - docker buildx build --platform linux/arm64 --tag ${LOCAL_IMAGE} --push . only: - tags ```

QEMU is a popular open-source emulator that can emulate a variety of CPU architectures, including ARM, PowerPC, and MIPS. multiarch/qemu-user-static is a Docker image that allows you to emulate different CPU architectures on your host machine using QEMU.

Next, you need to enable Qemu support for your Docker host instance. Qemu is a generic machine emulator and virtualiser. In simple terms, it allows your machine with one type of processor to emulate other types of processor, allowing your machine to execute scripts on different processor architectures.

Check out https://github.com/multiarch/qemu-user-static.

Another great tool for cross arch development is qemu-static-user. It allows you to run your nonnative architecture docker images on host using a qemu container for translation and only requires about one command to get working. It’s life saving for me when I need to test but setting up a full VM or native server is too time consuming.

https://github.com/multiarch/qemu-user-static

I'm just gonna leave these 2 here since I think they take a lot less effort to setup than the gcc-cross stuff and should be able to help you do stuff a lot faster: https://wiki.debian.org/QemuUserEmulation https://github.com/multiarch/qemu-user-static

multiarch/qemu-user-static is required for emulating ARM32 and Aarch64 instructions on x86_64 Linux.

This is MacOS's user mode x86 emulator ported to Linux. They intend for Linux VMs to use it as the interpreter for x86 binaries, including in containers. Basically a faster replacement for qemu-user-static .

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.