enhancements VS connaisseur

Various pieces support pieces for pod to pod mTLS are slowly being brought into the main Kubernetes project.

Take a look at https://github.com/kubernetes/enhancements/tree/master/keps/..., which is hopefully landing as alpha in Kubernetes 1.34. It lets you run a controller that issues certificates, and the certificates get automatically plumbed down into pod filesystems, and refresh is handled automatically.

Together with ClusterTrustBundles (KEP 3257), these are all the pieces that are needed for someone to put together a controller that distributes certificates and trust anchors to every pod in the cluster.

More information can be found in the Kubernetes Enhancement Proposal (KEP) for in-place resizing.

Digging into the Kubernetes source code led me to KEP-4633: Make anonymous authentication configuration endpoints configurable. This KEP addresses the exact concern of wanting to disable anonymous access globally while still allowing essential health checks (which don't necessarily need full TCP checks to be useful).

Kubernetes 1.33, scheduled for late April, introduces kuberc, a new feature that allows you to customize kubectl. This feature, described in KEP-3104, enables you to create aliases and enforce options for kubectl subcommands.

Might be noteworthy that in recent enough k8s the lifecycle.preStop.sleep.seconds is implemented https://github.com/kubernetes/enhancements/blob/master/keps/...

In September of 2016 the enhancement #88 (KEP) issue is created to support out-of-tree cloud providers (pluggable).

The motivation is more the latter, but it's not at all clear the proposed removal of the embedded kustomize will proceed, given the compatibility implications. See discussion at https://github.com/kubernetes/enhancements/issues/4706#issue... and following.

(I do see there are some proposed enhancements related to profiles that might help here)

Thanks for these links!

I picked out one at random just to check if my skeptical reaction is fair: https://github.com/kubernetes/enhancements/tree/master/keps/...

- OK, this is actually a really good and useful doc!

- However, it's not an up-front design doc, it has clearly been written after the bulk of the work has been done, to explain and justify rolling out a big change. (See the "implementation history" timeline: https://github.com/kubernetes/enhancements/tree/master/keps/...)

- It looks like the template wasn't very useful; most of the required sections are marked "N/A", and there are comments like The best test for work like this is, more or less, "did it work?"

> was always told early on that although they supported vault on kubernetes via a helm chart, they did not recommend using it on anything but EC2 instances (because of "security" which never really made sense their reasoning).

The reasoning is basically that there are some security and isolation guarantees you don't get in Kubernetes that you do get on bare metal or (to a somewhat lesser extent) in VMs.

In particular for Kubernetes, Vault wants to run as a non-root user and set the IPC_LOCK capability when it starts to prevent its memory from being swapped to disk. While in Docker you can directly enable this by adding capabilities when you launch the container, Kubernetes has an issue because of the way it handles non-root container users specified in a pod manifest, detailed in a (long-dormant) KEP: https://github.com/kubernetes/enhancements/blob/master/keps/... (tl;dr: Kubernetes runs the container process as root, with the specified capabilities added, but then switches it to the non-root UID, which causes the explicitly-added capabilities to be dropped).

You can work around this by rebuilding the container and setting the capability directly on the binary, but the upstream build of the binary and the one in the container image don't come with that set (because the user should set it at runtime if running the container image directly, and the systemd unit sets it via systemd if running as a systemd service, so there's no need to do that except for working around Kubernetes' ambient-capability issue).

> It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."

Another thing to consider with image signatures is how you will be running the containers. For most of us, Kubernetes is our platform of choice, and it does not have native support for DCT, so unless you are using a specific distribution that implements it, you are going to need to provide some form of runtime enforcement. Fortunately, the Kubernetes admission controller API can be leveraged to do this, and open source projects like Connaisseur can take care of this for DCT / Notary v1 as well as Cosign signatures.

We already mentioned Connaisseur Admission Controller as a way to enforce content trust and reject images that are not signed by trusted sources.

Admission Controller was based on Connaisseur, heavily modified to work with v2 instead of v1 signatures.