Vagrant VS QEMU

Install Vagrant: Download and install Vagrant.

Ensuring high availability with limited resources can be challenging. I recently want to proove you can do it using Keepalived and just two servers 💪✨. To prove it, I used Vagrant. Here's a quick rundown of my journey! 🚀

Vagrant - Make sure you have Vagrant installed on your machine. You can download it from Vagrant's official site.

I do believe that using containers makes a ton of sense in writing data pipelines. You can use the same image to develop and run the pipeline, preventing "it works on my machine" issues. You can test different variations of the image without having to stand up additional infrastructure or potentially breaking the workflows of others who're using the same infrastructure. Finally, knowledge of containerization is increasingly expected of all engineers, while knowledge of other tools that solve similar issues (like Vagrant or Ansible) is less common.

Running NixOS on a virtual machine (VM) is a safe and reproducible way to test such configurations. As for VMs, I have used VirtualBox, Vagrant and lxd in the past. However, I have found QEMU to be the simplest and most flexible solution for my needs.

I found out that the error was that initialy I was using an old version of the vagrant utility for vmware: 1.0.21, so I read this post: https://github.com/hashicorp/vagrant/issues/12052 and there I saw that I needed to install the new version, in my case 1.0.22 which I mentioned on the point #3.

Vagrant is a tool for building and managing virtual machine environments in a single workflow. Developed by Hashicorp, it is used to replicate multiple virtual environments. It can efficiently run in all virtualized environments, providing the highest level of isolation to users.

Another important tool from the same organization is Vagrant, which provides extra help in running VMs built with Packer. Of course, the choice of a VM provider is also very important, as some VM providers may not be supported on certain platforms. For example, there are no VMware or VirtualBox releases that support Apple Silicon. However, QEMU is supported on most platforms, including Apple Silicon, which is why this provider was chosen here.

QEMU with LibVirt GUI - what I will be using. An excellent choice for an existing Linux installation, does what it should, and no more, gets out of your way.

QEMU is a generic and open-source machine emulator and virtualizer.

That is expected: We cannot run an ARM-based Docker image on an x86_64 host without some additional setup, in particular, using QEMU.

Using Docker to run Tailwind in an arm64 container via QEMU emulation (proved too slow)

My original solution was an obvious one but had an interesting bug. I simply would use Perl's system command to execute the SlackBuild script with a shell, where I would pipe stdout to tee, writing the stdout to a temporary file. After the SlackBuild would execute I would read the temporary file looking for the Slackware package $PATH created line. This solved the problem of allowing both the user and sbozyp to read the stdout of the SlackBuild script. For 99% of builds this worked fine ... but then there was qemu. Qemu is a massive project that requires a huge compilation. My system uses a small 4GB tmpfs mounted at /tmp, and the temporary file that was being teed to ended up getting so large from all the qemu compilation output that tee (and thus sbozyp) crashed for "device out of space".

There are plenty of tutorials and other Docker images running Raspberry Pi OS using QEMU, but unfortunately, all of them utilize the OS image at runtime as an SD card. Hence, they do not support mounting volumes to share the filesystem.

There are several choices for running code in partial or full isolation. Some languages include lightweight environments that do not interfere with each other, e.g., virtual environments in Python. However, due to caching and links, these are not sufficiently isolated for us. At the other end of the spectrum, we can run code in a node of a cloud computing service. However, the overhead and cost make this not worthwhile given our needs: isolation, but not very strong security requirements. Alternatively, we can run a virtual machine or emulator such as QEMU, VirtualBox, or others. This also has too much overhead given our needs.