enhancements VS cri-dockerd

> was always told early on that although they supported vault on kubernetes via a helm chart, they did not recommend using it on anything but EC2 instances (because of "security" which never really made sense their reasoning).

The reasoning is basically that there are some security and isolation guarantees you don't get in Kubernetes that you do get on bare metal or (to a somewhat lesser extent) in VMs.

In particular for Kubernetes, Vault wants to run as a non-root user and set the IPC_LOCK capability when it starts to prevent its memory from being swapped to disk. While in Docker you can directly enable this by adding capabilities when you launch the container, Kubernetes has an issue because of the way it handles non-root container users specified in a pod manifest, detailed in a (long-dormant) KEP: https://github.com/kubernetes/enhancements/blob/master/keps/... (tl;dr: Kubernetes runs the container process as root, with the specified capabilities added, but then switches it to the non-root UID, which causes the explicitly-added capabilities to be dropped).

You can work around this by rebuilding the container and setting the capability directly on the binary, but the upstream build of the binary and the one in the container image don't come with that set (because the user should set it at runtime if running the container image directly, and the systemd unit sets it via systemd if running as a systemd service, so there's no need to do that except for working around Kubernetes' ambient-capability issue).

> It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."

0 is not the request we've defined. And that makes sense. Memory QoS has been in alpha since Kubernetes 1.22 (August 2021) and according to the KEP data was still in alpha as of 1.27.

Note: There's actually a Structured Authentication Config established via KEP-3331. It's in v1.28 as a feature flag gated option and removes the limitation of only having one OIDC provider. I may look into doing an article on it, but for now I'll deal with the issue in a manner that should work even with a bit older versions versions of Kubernetes.

Kubernetes supports a skew policy of n+2 between API server and kubelet. This means if your CP and DP are both on 1.20, you could upgrade your control plane twice (1.20 -> 1.21 -> 1.22) before you need to upgrade your data plane. And when it comes time to upgrade your data plane you can jump from 1.20 to 1.22 to minimize update churn. In the future, this skew will be opened to n+3 https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/3935-oldest-node-newest-control-plane

The KEP (Kubernetes Enhancement Proposal) is linked to in the PR [1]. From the summary:

> Sidecar containers are a new type of containers that start among the Init containers, run through the lifecycle of the Pod and don’t block pod termination. Kubelet makes a best effort to keep them alive and running while other containers are running.

[1] https://github.com/kubernetes/enhancements/tree/master/keps/...

This is where the new feature of mutable scheduling directives for jobs comes into play. This feature enables the updating of a job's scheduling directives before it begins. Essentially, it allows custom queue controllers to influence pod placement without needing to directly handle the assignment of pods to nodes themselves. To learn more about this check out the Kubernetes Enhancement Proposal 2926.

What your asking is a (vanilla) Kubernetes non-goal, others have mentioned fluxcd and other add ons that provide primitives for dependency aware deployments. The problem space is so large, that it's unreasonable to to address these concerns in Kubernetes itself, instead, make it extensible... Look at this KEP for example: https://github.com/kubernetes/enhancements/issues/753 Sidecar containers have existed, and been named as such since WAY before that KEP's inception, defining what these things should and shouldn't do is largely arbitrary. Aka: your use-case is niche, if you don't like the behavior, use flux or argo, or write something yourself.

If further interested, I may recommend checking out the KEP. I love how they document the decision making, and all these edge cases :).

See https://kubernetes.io/docs/reference/kubectl/conventions/#subresources and https://github.com/kubernetes/enhancements/issues/2590

$ wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.9/cri-dockerd-0.3.9.amd64.tgz $ tar zxvf cri-dockerd-0.3.9.amd64.tgz $ cd cri-dockerd $ sudo mkdir -p /usr/local/bin $ sudo install -o root -g root -m 0755 cri-dockerd/usr/local/bin/cri-dockerd $ mkdir foo; cd foo $ git clone [email protected]:Mirantis/cri-dockerd.git $ cd cri-dockerd $ sudo install packaging/systemd/* /etc/systemd/system $ sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service $ sudo systemctl daemon-reload $ sudo systemctl enable --now cri-docker.socket

I installed cri-dockerd using this documentation https://github.com/Mirantis/cri-dockerd

99% of what you just said is completely incorrect. containerd is not a facade for Docker, in fact the Docker engine is a facade for containerd. The OCI spec is also not a facade for Docker, Docker is simply one application which can create OCI compliant images which can be executed by runtimes like runc. Kubernetes has zero facades for Docker, unless you count the optional open-source cri-dockerd.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ Adjust netplan for static IP Adding br_netfilter and overlay to kernel Creating /etc/modules-load.d/k8s.conf with bridging and forwarding https://kubernetes.io/docs/setup/production-environment/container-runtimes/#forwarding-ipv4-and-letting-iptables-see-bridged-traffic Installing docker components https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker Installing cri-dockerd https://github.com/Mirantis/cri-dockerd Disabling swap https://graspingtech.com/disable-swap-ubuntu/ install kubeadm kubectl kubelet https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Im trying to learn how to set up kubernetes cluster with kubeadm. Im using official kubernetes documentation ( maybe there is a better source?). My goal is to set it up with cri-dockerd and systemd. And to be honest its quite hard task to do it. Informations are scattered arround with links and sometimes its hard to know the order which steps should be executed. I have 2 Virtualbox machines, connected together with NAT network. A master and worker node. I performed these steps on both of them: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ Adjust netplan for static IP Adding br_netfilter and overlay to kernel Creating /etc/modules-load.d/k8s.conf with bridging and forwarding https://kubernetes.io/docs/setup/production-environment/container-runtimes/#forwarding-ipv4-and-letting-iptables-see-bridged-traffic Installing docker components https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker Installing cri-dockerd https://github.com/Mirantis/cri-dockerd Disabling swap install kubeadm kubectl kubelet https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ configure kubelet cgroup driver https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/ So currently im at the configuring cgroup driver. I tried to execute kubeadm init kubeadm-config.yaml but kubeadm found 2 CRI endpoints. So i tried to point the correct one with this command kubeadm join --cri-socket /var/run/cri-dockerd.sock but i got discover:Invalid value: ::: bootstrapToken or file must be set And now im completely lost. And also im confused is this a moment for installing CNI (probably i will try with Calico) for pod communication or should i start the cluster first?

But not to worry, you can still use Docker as a container runtime in Kubernetes using the cri-dockerd adapter. cri-dockerd provides a shim for Docker Engine that lets you control Docker via the Kubernetes CRI. ****

As of version 1.24, either one of the other supported runtimes (e.g. containerd or CRI-O) or, in case you still want to rely on Docker Engine, cri-dockerd must be used. Further information on precautions that may be necessary due to the removal of Dockershim is provided by Kubernetes in a guide.

Kubernetes requires a tool which implements the Container Runtime Interface, a standardized API for starting & managing containers. This is from 2015-2016[1].

For a while Kubernetes has included something called the "dockershim", it's own implementation of a CRI interface that, under the hood, calls Docker or Podman. There's also tools like kind[2] ("kubernetes in docker") that go further- not just hosting Kubernetes worker containers in Docker, but hosting the main kubernetes daemons also in Docker.

Kubernetes deprecated Dockershim, formally in December 2020, but is just throwing the switch now in the upcoming 1.24, expected mid-April[3]. A company Mirantis has pledged to take over support of Dockershim[4], and is calling the new effort "cri-dockerd"[5]. This should allow Kubernetes workers to continue to run via Docker or Podman.

Kind is unaffected, since it runs the main Kubernetes controllers in Docker, which then launch their own opencontainerd (one off the main CRI implementations) inside that Docker container, nested like, so no dockership/cri-dockerd is needed).

[1] https://kubernetes.io/blog/2016/12/container-runtime-interfa...

[2] https://kind.sigs.k8s.io/

[3] https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-o...

[4] https://www.mirantis.com/blog/mirantis-to-take-over-support-...

[5] https://github.com/Mirantis/cri-dockerd