Nomad VS Docker Compose

20k+ nodes and 200k+ allocs. To be fair, Kubernetes cannot support this large of a cluster.

Most of my issues with it aren't related to the scale though. I wasn't involved in the operations of the cluster, I was just a user of Nomad trying to run a few thousand stateful allocs. Without custom resources and custom controllers, managing stateful services was a pain in the ass. Critical bugs would also often take years to get fixed. I had lots of fun getting paged in the middle of the night because 2 allocs would suddenly decide they now have the same index (https://github.com/hashicorp/nomad/issues/10727)

https://github.com/hashicorp/nomad/blob/v1.6.0/website/conte... seems to have existed since before the license rug-pull. However I'm open to there being some miscommunication because https://developer.hashicorp.com/nomad/docs/glossary doesn't mention the word "control" and the word "host" could mean any number of things in this context

Nomad: The Oasis in Our Infrastructure Desert

> I should have just used on-disk mode from the start, but only now know better.

Yeah, I saw the recent post about reducing rqlite disk space usage. Using the on-disk sqlite as both the FSM and the Raft snapshot makes a lot of sense here. I'm curious whether you've had concerns about write amplification though? Because we have only the periodic Raft snapshots and the FSM is in-memory, during high write volumes we're only really hammering disk with the Raft logs.

> Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite.

My understanding is that rqlite Raft entries are mostly SQL statements (is that right?). Where Nomad is somewhat different (and probably closer to the OP) is that the Raft entries are application-level entries. For entries that are commands like "stop this job"[0] upgrades are simple.

The tricky entries are where the entry is "upsert this large deeply-nested object that I've serialized", like the Job or Node (where the workloads run). The typical bug here is you've added a field way down in the guts of one of these objects that's a pointer to a new struct. When old versions deserialize the message they ignore the new field and that's easy to reason about. But if the leader is still on an old version and the new code deserializes the old object, you need to make sure you're not missing any nil pointer checks. Without sum types enforced at compile time (i.e. Option/Maybe), we have to catch all these via code review and a lot of tedious upgrade testing.

> it requires discipline on the part of the end-users too.

Oh for sure. Nomad runs into some commercial realities here around how much discipline we can demand from end-users. =)

[0] https://github.com/hashicorp/nomad/blob/v1.8.2/nomad/fsm.go#...

However, Mesos has a steeper learning curve compared to Kubernetes and may require more expertise to manage effectively. For those seeking a simpler but less feature-rich alternative, HashiCorp's Nomad is worth considering.

Just wanting to understand the definition of Open Source you are using here, considering that you can see and download the source here:

https://github.com/hashicorp/nomad

Are you maybe referring to their Enterprise offering?

Even though this article will be focused on Kubernetes I want to mention that there are multiple container orchestration platforms such as Mesos, Docker Swarm, OpenShift, Rancher, Hashicorp Nomad, etc.

Step 4:Install Docker Compose: Now we install Docker Compose using the command: sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" \ -o /usr/local/bin/docker-compose

$ sudo yum install docker git -y $ sudo systemctl enable --now docker $ # Attention: You need IPv4 to download from GitHub! $ sudo curl -L \ "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" \ -o /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose $ sudo usermod -aG docker $(whoami) # Log in to the instance again

[ec2-user]$ sudo yum update -y [ec2-user]$ sudo yum install -y docker [ec2-user]$ sudo service docker start [ec2-user]$ sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose [ec2-user]$ sudo chmod +x /usr/local/bin/docker-compose [ec2-user]$ docker --version Docker version 20.10.23, build 7155243 [ec2-user]$ docker-compose --version Docker Compose version v2.18.1

If you're a fan of compose files, then you can use Docker's own compose application [1] with Podman [2]. It seems that the compose cli controls the engine using its socket. Both podman and docker engines have the almost same API. I'm using this approach since podman-compose didn't work as expected for me. Docker-compose is usually installed as a plugin for the docker client. However, I use it as a standalone application for use with Podman. In addition, I prefer using docker 'contexts' instead of the DOCKER_HOST environment variable to set up the integration.

Another relevant project is podlet [3]. It can be used to convert compose files, podman/docker command or a running object to a set of quadlet files. Also, note that if plain quadlets aren't powerful enough for you, quadlets [4] and plain podman [5] also support running a limited set of kubernetes manifests.

[1] https://github.com/docker/compose

[2] https://docs.podman.io/en/latest/markdown/podman-system-serv...

[3] https://github.com/containers/podlet

[4] https://docs.podman.io/en/latest/markdown/podman-systemd.uni...

[5] https://docs.podman.io/en/latest/markdown/podman-kube-play.1...

name: CD Pipeline on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v3 - name: Deploy via SSH uses: appleboy/ssh-action@master with: host: ${{ secrets.SSH_HOST }} username: ${{ secrets.SSH_USERNAME }} key: ${{ secrets.SSH_PRIVATE_KEY }} script: | # Update package index and install dependencies sudo apt-get update -y sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common # Add Docker's official GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # Install Docker sudo apt-get update -y sudo apt-get install -y docker-ce docker-ce-cli containerd.io # Add the SSH user to the Docker group sudo usermod -aG docker ${{ secrets.SSH_USERNAME }} # Install Docker Compose sudo curl -L "https://github.com/docker/compose/releases/download/v2.20.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose # Verify installations docker --version docker-compose --version # Navigate to the project directory and deploy cd /home/ubuntu/ git clone cd fastapi-book-project/ git pull docker-compose up -d --build

#!/bin/bash sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo -i chmod +x /usr/local/bin/docker-compose docker compose version