kubelogin VS k3s

Authenticating with Kyma is a (in my opinion) unnecessary challenge as it leverages the OIDC-login plugin for kubectl. You find a description of the setup here. This works fine when on a Mac but can give you some headaches on a Windows and on Linux machine especially when combined with restrictive setups in corporate environments. For Windows I can only recommend installing krew via chocolatey and then install the OIDC plugin via kubectl krew install oidc-login. At least for me that was the only way to get this working on Windows.

It is sort of a roundabout way, but I sync Active Directory to a Keycloak realm, then use OIDC auth with kube-oidc-proxy (https://github.com/jetstack/kube-oidc-proxy) and kubelogin (https://github.com/int128/kubelogin) for OIDC-based auth to the api server.

Yes, I setup a cluster with no SPFs. That means an HA setup for the external load balancer. I use HAProxy for my ELB, and setup 2 instances with a VRRP + keepalived to provide HA to the ingress controller. I run the control plane private, accessible only from localhost. I setup kube-oidc-proxy (https://github.com/jetstack/kube-oidc-proxy) to expose the API server with single sign-on on the ingress controller, and use the kubelogin plugin (https://github.com/int128/kubelogin) to provide OIDC support to kubectl. I then setup Keycloak to handle OIDC/OAuth2/SAML and syncing to Active Directory, and setup groups in Active Directory to control acccess to clusters. Devs each get their own namespace in the dev cluster, with mostly cluster-admin access to their namespace. Staging/Prod clusters are locked down, with read-only access to devs. Thanks to the OIDC auth to the APIServer, when employees are onboarded & offboarded, we only need to add/remove them from groups in Active Directory and everything else just magically syncs.

Link to GitHub Repository

kubelogin (helper for k8s build in OIDC support)

I use the kube-login plugin for kubectl (https://github.com/int128/kubelogin) along with the kube-oidc-proxy (https://github.com/jetstack/kube-oidc-proxy), using Keycloak as my OIDC provider (https://www.keycloak.org) and doing LDAP synchronization to Active Directory.

With a working dex/OIDC configuration, you could use: https://github.com/int128/kubelogin

We use Keycloak for this purpose. We deploy an OIDC-proxy to the kube-api (https://github.com/jetstack/kube-oidc-proxy), then use the kubectl plugin 'kubelogin' (aka oidc-login if you use krew - https://github.com/int128/kubelogin). This gives us the ability to have no user secrets in our KUBECONFIG, and to use Keycloak's Active Directory/LDAP user & group federation to control access to clusters. With this, downloading the KUBECONFIG is self-service, and adding users to new clusters is as easy as adding them to a group in AD.

Before we can go ahead and test this out, we need to do some setup for kubectl so that it knows how to do OIDC authentication. We need to install kubelogin plugin for this. Go ahead and install it using any of the following commands.

kubelogin - If you're using OIDC provider such as Google, Keycloak or Dex for authenticating to Kubernetes cluster, then this plugin also known as oidc-login in krew can help you avoid having to manually login into your cluster over and over again. When you setup this plugin, every time you attempt to run any kubectl command without having valid authentication token, oidc-login will automatically open your provider's login page and after successful authentication grabs the token and logs you into the cluster. To see video of workflow check out the repository here.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.

K3s' go.mod[0] is insane.

[0] https://github.com/k3s-io/k3s/blob/master/go.mod