nerdctl VS registry.k8s.io

What about the docker-cli? colima also ships with a docker-compatible cli to interact with containerd called nerdctl. We can execute the same docker cli commands like:

Using nerdctl: https://github.com/containerd/nerdctl

I'd really disagree that compose files are somehow one-shot, or blindly modified. To the contrary, really, we have them checked in with the source code. Upon deployment to the cluster, the (running) services will be intelligently updated or replaced (in a rolling manner, causing zero downtime). LXC might be more elegant, but I have no idea what simple, file-based format I could use to let engineers describe the environment their app should run in without compose.

I need something that even junior devs can start up with a single command, that can be placed in the VCS along with the code, and that will not require deep Linux knowledge to get running. Open for suggestions here, really.

Now since Kubernetes works off of containerd I'll be taking a different approach on handling container builds by using nerdctl and the buildkit that comes bundled with it. I'll do this on the amd64 control plane node since it's beefier than my Raspberry Pi workers for handling builds and build related services. Go ahead and download and unpack the latest nerdctl release as of writing (make sure to check the release page in case there's a new one):

That's why nerdctl, their cli binary, is so well named.

Well, it is indeed a good option. However, containerd is a good alternative that is growing even among developers. Please see: https://github.com/containerd/nerdctl

Nerdctl/containerd has IPFS support :)

https://github.com/containerd/nerdctl/blob/main/docs/ipfs.md

nerdctl supports IPFS for both image pulling and pushing, including encrypted images and eStargz lazy pulling. For building, the current method is a locally hosted translator so that the traditional pulls can be converted to work over IPFS. They even have docs on running it on k8s node, though if my reading is correct this isn't exactly a cloud native approach (running systemd services on each node...).

If you stop to think about, what AWS is selling here, it's quite funny. Most of the infrastructure behind registry.k8s.io is hosted on AWS (and also GCP). So AWS essentially tells you: Don't trust the upstream registry, it might go down, cache it on your own registry, hosted also by us.

For example: if you only allow cluster autoscaler and metrics server from registry.k8s.io you can pull those images through the cache as someone who has create repo IAM privileges. If someone without create repo privileges tries to pull a new image it will fail because they can't create the initial repo.

https://registry.k8s.io (the root url at /) redirects you to https://github.com/kubernetes/registry.k8s.io where we have an issue tracker.

❗ This container is having trouble accessing https://registry.k8s.io

Hosting a forwarding / redirect server instead of actually hosting images is probably a decent idea.

The K8s proxy is redirecting from only hosting on GCR to community-owned registries - https://kubernetes.io/blog/2023/03/10/image-registry-redirec...

You can view the code here - https://github.com/kubernetes/registry.k8s.io

But because everyone is already pointing at gcr.io (just like many openfaas users point at docker.io/) - they're having to do a huge campaign to announce the new URL - the same would apply with the author's solution here.

I wrote some automation for hosting (not redirects) in arkade with the OSS registry - Get a TLS-enabled Docker registry in 5 minutes - https://blog.alexellis.io/get-a-tls-enabled-docker-registry-...

The registry is also something you can run on a VM if you so wish, and have act as a pull through cache.

Apart from reliability - GitHub's container registry is the current next best option - but we have to ask ourselves, what happens when they start charging or the outages start to last longer or are more frequent than 1-2 times per week as we've seen in Q1 2023.

One annoyance with how docker images are specified is they include the location where they are stored. So if you want to change where you store you image you break everyone.

I wonder if what regsitry.k8s.io does could be generalized:

https://github.com/kubernetes/registry.k8s.io/blob/main/cmd/...

The idea is the depending on which cloud you are pulling the image from, they will use the closest blob store to service the request. This also has the effect that you could change the source of truth for the registry without breaking all Dockerfiles.

If you are using updated helm charts then most of them have already replaced with registry.k8s.io, for example nginx ingress so not really breaking change.