dive VS buildkit

[Source: Dive on GitHub]

Learn more on the Dive GitHub page.

curl -L https://github.com/wagoodman/dive/releases/download/v0.9.2/dive_0.9.2_linux_amd64.tar.gz -o dive.tar.gz tar -xvf dive.tar.gz

Cool, gonna try this soon. Would be great to use in combination with Dive (https://github.com/wagoodman/dive)

Dive is an open-source tool that allows you to explore the various layers of a Docker image. It shows you the content of each layer and helps you identify voluminous or unnecessary parts.

Dive is a great tool for debugging this. I like image reduction work just because it gives me a chance to play with Dive: https://github.com/wagoodman/dive

One easy low hanging fruit I see a LOT for ballooning image sizes is people including the kitchen sink SDK/CLI for their cloud provider (like AWS or GCP), when they really only need 1/100 of that. The full versions of both of these tools are several hundred mb each

BuildKit GitHub Repository

I remember when container builds were slow and not really efficient, and were usually a bottleneck of our CI/CD pipelines. That is until I discovered BuildKit and my life changed. BuildKit is the next-generation builder engine for Docker, but it can also be used independently.

BuildKit is the engine behind Docker's build system. It's a fascinating look into how Docker images are constructed, with features like:

create should be provided with a daemon configuration file through the --buildkitd-config flag (if not, it defaults to the buildkitd.default.toml file contained in the config directory of buildx). You can find an example of a complete configuration file in buildkit official documentation on GitHub.

By default, Kamal uses the docker-container driver to build images which, in turn, uses the BuildKit toolkit internally. While Kamal sets up registry caching correctly, caching still fails in the end because the BuildKit process is isolated from our GitHub Action runtime process. To connect the two, we need to expose the GitHub runtime to the workflow. Luckily, there is a GitHub Action ready just for this so all that is needed is adding the action to the workflow file. We put it right after setting up Docker Buildx:

## Local Cache This method leverages local storage for caching Docker layers. The downside is that old cache entries aren’t deleted automatically, so the cache size might increase over time. A temporary fix involves moving and cleaning the cache after each build. ```yaml //".github/workflows/build.yml" name: Docker Build on: push: jobs: docker: runs-on: ubuntu-latest steps: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 // highlight-start - name: Cache Docker layers uses: actions/cache@v4 with: path: /tmp/.buildx-cache key: ${{ runner.os }}-buildx-${{ github.sha }} restore-keys: | ${{ runner.os }}-buildx- // highlight-end - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ vars.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} // highlight-start - name: Build and push uses: docker/build-push-action@v6 with: push: true tags: user/app:latest cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max // highlight-end - # Temp fix # https://github.com/docker/build-push-action/issues/252 # https://github.com/moby/buildkit/issues/1896 name: Move cache run: | rm -rf /tmp/.buildx-cache mv /tmp/.buildx-cache-new /tmp/.buildx-cache

At the very real risk of talking out of my ass, the new versioned Dockerfile mechanism on top of builtkit should enable you to do that: https://github.com/moby/buildkit/blob/v0.15.0/frontend/docke...

In true "when all you have is a hammer" fashion, as very best I can tell that syntax= directive is pointing to a separate docker image whose job it is to read the file and translate it into builtkit api calls, e.g. https://github.com/moby/buildkit/blob/v0.15.0/frontend/docke...

But, again for clarity: I've never tried such a stunt, that's just the impression I get from having done mortal kombat with builtkit's other silly parts

The currently proposed solution is to allow Docker to bind the cache directory in the build to a directory on the host. This way the cache could be persisted externally. However, this issue has been opened for almost 4 years (May 27, 2020) with no clear answer as to whether it'll be implemented any time soon.

> We are uding docker-in-docker at the moment

You can also run a "less privileged" container with all the features of Docker by using rootless buildkit in Kubernetes. Here are some examples:

https://github.com/moby/buildkit/tree/master/examples/kubern...

https://github.com/moby/buildkit/blob/master/examples/kubern...

It's also possible to run dedicated buildkitd workers and connect to them remotely.