buildkit VS buildx

BuildKit Faster Docker Builds Build speed + cache control = win.

Windows is is very similar, the differences are two the layer tarballs.

The file system appears in a Files sub-directory as there is a Hives sub-directory for containing the Windows Registry.

The other difference is there are two extra PAX headers within the tarball, MSWINDOWS.fileattr which is "32" for a regular file, and "16" for a directory and MSWINDOWS.rawsd which is a special encoding of the security descriptor, which you can think of it as the owner, group and permissions associated with the file (which their standard values can be seen from buildkit here: https://github.com/moby/buildkit/blob/22156ab20bcaea1a1466d2...)

I haven't looked into how to handle the Windows Registry aspect as in my exploration I was focused on simply adding a pre-built executable so I didn't need any registry entries created.

The other fun gotcha is to ensure the ENV section contain PATH set to c:\\Windows\\System32;c:\\Windows otherwise you would be unlikely to be able to run any Windows executable.

BuildKit GitHub Repository

I remember when container builds were slow and not really efficient, and were usually a bottleneck of our CI/CD pipelines. That is until I discovered BuildKit and my life changed. BuildKit is the next-generation builder engine for Docker, but it can also be used independently.

BuildKit is the engine behind Docker's build system. It's a fascinating look into how Docker images are constructed, with features like:

create should be provided with a daemon configuration file through the --buildkitd-config flag (if not, it defaults to the buildkitd.default.toml file contained in the config directory of buildx). You can find an example of a complete configuration file in buildkit official documentation on GitHub.

By default, Kamal uses the docker-container driver to build images which, in turn, uses the BuildKit toolkit internally. While Kamal sets up registry caching correctly, caching still fails in the end because the BuildKit process is isolated from our GitHub Action runtime process. To connect the two, we need to expose the GitHub runtime to the workflow. Luckily, there is a GitHub Action ready just for this so all that is needed is adding the action to the workflow file. We put it right after setting up Docker Buildx:

## Local Cache This method leverages local storage for caching Docker layers. The downside is that old cache entries aren’t deleted automatically, so the cache size might increase over time. A temporary fix involves moving and cleaning the cache after each build. ```yaml //".github/workflows/build.yml" name: Docker Build on: push: jobs: docker: runs-on: ubuntu-latest steps: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 // highlight-start - name: Cache Docker layers uses: actions/cache@v4 with: path: /tmp/.buildx-cache key: ${{ runner.os }}-buildx-${{ github.sha }} restore-keys: | ${{ runner.os }}-buildx- // highlight-end - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ vars.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} // highlight-start - name: Build and push uses: docker/build-push-action@v6 with: push: true tags: user/app:latest cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max // highlight-end - # Temp fix # https://github.com/docker/build-push-action/issues/252 # https://github.com/moby/buildkit/issues/1896 name: Move cache run: | rm -rf /tmp/.buildx-cache mv /tmp/.buildx-cache-new /tmp/.buildx-cache

At the very real risk of talking out of my ass, the new versioned Dockerfile mechanism on top of builtkit should enable you to do that: https://github.com/moby/buildkit/blob/v0.15.0/frontend/docke...

In true "when all you have is a hammer" fashion, as very best I can tell that syntax= directive is pointing to a separate docker image whose job it is to read the file and translate it into builtkit api calls, e.g. https://github.com/moby/buildkit/blob/v0.15.0/frontend/docke...

But, again for clarity: I've never tried such a stunt, that's just the impression I get from having done mortal kombat with builtkit's other silly parts

The currently proposed solution is to allow Docker to bind the cache directory in the build to a directory on the host. This way the cache could be persisted externally. However, this issue has been opened for almost 4 years (May 27, 2020) with no clear answer as to whether it'll be implemented any time soon.

The client can also be SDK, although the SDKs are for the core Docker and not for the plugins like Buildx.

To install the newest builder, download the latest version of the buildxplugin by visiting https://github.com/docker/buildx/releases. Find the most recent release, go to the “Assets” section, and expand the list of assets. Download the file that ends with windows-amd64.exe, like buildx-v0.14.0.windows-amd64.exe. Place the downloaded file in the cli-plugins folder you created earlier, and rename it to docker-buildx.exe. Now, the command docker build will automatically use the new builder.

This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104

Docs: https://docs.docker.com/engine/reference/commandline/buildx_...

Container engine: Docker v23.0.4 | Buildx v0.10.4

ARCH=amd64 # change to 'arm64' for M1 VERSION=v0.10.4 curl -LO https://github.com/docker/buildx/releases/download/${VERSION}/buildx-${VERSION}.darwin-${ARCH} mkdir -p ~/.docker/cli-plugins mv buildx-${VERSION}.darwin-${ARCH} ~/.docker/cli-plugins/docker-buildx chmod +x ~/.docker/cli-plugins/docker-buildx docker buildx version # verify installation

wget -O /tmp/docker/cli-plugins/docker-buildx https://github.com/docker/buildx/releases/tag/v0.10.4/buildx-v0.10.4.linux-amd64

Your particular one looks somewhat like https://github.com/docker/buildx/issues/1595.

Now, we have examples of the actual definitions for a single application’s docker container. Notice that we set the contexts key here which references the dependency targets we just defined. You can basically think of this like the depends_on block if you’ve used Terraform before. The args key let’s us populate the ARG variables in the Dockerfile. This is what is ultimately different between each of the containers. In the future, the buildx team might support using for_each loops like in Terraform but for now each block will have a bit of duplication.

Now as bonus we i will add how to use this techniques to build a multi arch build container. Maybe you have the same application but you need to run it in arm or riscv architecture. For this we can use the buildx plugin from docker https://github.com/docker/buildx.