cert-manager VS k3s

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.25.8+k3s1 INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --disable=traefik --cluster-cidr=10.42.0.0/16" sh - Install kubectl from APT https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/ Install helm from APT https://helm.sh/docs/intro/install/ cp /etc/rancher/k3s/k3s.yaml .kube/config cp /etc/rancher/k3s/k3s.yaml /root/.kube/config kubectl create -f tigera-operator.yaml #Change ippools CIDR to 10.42.0.0/16 kubectl create -f custom-resources.yaml watch kubectl get pods --all-namespaces kubectl get nodes -o wide helm repo add rancher-stable https://releases.rancher.com/server-charts/stable kubectl create namespace cattle-system kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.1 helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=scrapper.todoroff.net --set global.cattle.psp.enabled=false kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' xxxxxxxxxxxxxxv6h72ckxp2xz2fpgqrlw864s2wjxbw8mwcr75

root@edgevpn1:~# kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created namespace/cert-manager created serviceaccount/cert-manager-cainjector created serviceaccount/cert-manager created serviceaccount/cert-manager-webhook created clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrole.rbac.authorization.k8s.io/cert-manager-view created clusterrole.rbac.authorization.k8s.io/cert-manager-edit created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created role.rbac.authorization.k8s.io/cert-manager:leaderelection created role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created service/cert-manager created service/cert-manager-webhook created deployment.apps/cert-manager-cainjector created deployment.apps/cert-manager created deployment.apps/cert-manager-webhook created mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

I am using cert-manager 0.5.2 to manage Let's Encrypt certificates on our Kubernetes cluster.

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest helm repo add jetstack https://charts.jetstack.io kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.crds.yaml helm upgrade -i cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace helm upgrade -i rancher rancher-latest/rancher --version 2.5.4 --create-namespace --namespace cattle-system --set hostname=rancher.zbs.local --set bootstrapPassword=bootStrapAllTheThings --set replicas=1

Waiting for http-01 challenge propagation: failed to perform self check GET request, it's similar to this bug https://github.com/jetstack/cert-manager/issues/656but all solutions from GitHub ticket comments didn't help.

- name: Finalize cluster become: no hosts: control[0] remote_user: ansibleadmin vars: tfconfig: "{{ lookup('file', 'variables.auto.tfvars.json') }}" tasks: - name: Remove taint to allow workloads on masters command: kubectl taint nodes --all=true node-role.kubernetes.io/master:NoSchedule- - name: Install helm ansible.builtin.shell: creates: /usr/local/bin/helm cmd: curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - name: Install Rancher Helm repo command: helm repo add rancher-stable https://releases.rancher.com/server-charts/stable - name: Install jetstack Helm repo command: helm repo add jetstack https://charts.jetstack.io - name: Update helm repos command: helm repo update - name: Create rancher namespace command: kubectl create namespace cattle-system - name: Install cert manager CRD command: kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml - name: Install cert manager Helm chart command: helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.7.1 - name: Install Rancher via Helm command: helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname={{ tfconfig['cluster-id'] }}-lb.{{ tfconfig['domain'] }} --set replicas=3 --set bootstrapPassword={{ tfconfig['rancher-password'] }} - name: Show boot pass debug: msg: "Visit this link in your browser to complete rancher setup: https://{{ tfconfig['cluster-id'] }}-lb.{{ tfconfig['domain'] }}/dashboard/?setup={{ tfconfig['rancher-password'] }}"

Opinionated meaning it picks, install, patches your CNI/Ingress/Load Balancer/DNS Server/Metrics Server/Monitoring Setup.

k3s is probably most well known as it ships with bunch of preinstall software: https://github.com/k3s-io/k3s so you can just start throwing yaml files at cluster and handling workloads. It's what I use for my homelab.

Paid things I've heard of include OpenStack and SideroLabs. Haven't used personally by SRE coworkers say good things about them.

For self-hosting I've found https://k3s.io to be really good from the SUSE people. Works on basically any Linux distro and makes self-hosting k8s not miserable.

Yes it’s going to depend on which k8s distribution you’re using. We have work in-progress for k3s to natively support nix-snapshotter: https://github.com/k3s-io/k3s/pull/9319

For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.

We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.

K3S: is a lightweight distribution of Kubernetes that is designed for resource-constrained environments. It is an excellent option for running Kubernetes on a virtual machine or cloud server.

I recently purchased a used Lenovo M900 Think Centre (i7 with 32GB RAM) from eBay to expand my mini-homelab, which was just a single Synology DS218+ plugged into my ISP's router (yuck!). Since I've been spending a big chunk of time at work playing around with Kubernetes, I figured that I'd put my skills to the test and run a k3s node on the new server. While I was familiar with k3s before starting this project, I'd never actually run it before, opting for tools like kind (and minikube before that) to run small test clusters for my local development work.

Kubernetes - k3s distribution

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.

You should be familiar with Kubernetes and have set up a Kubernetes cluster. I recommend k3s.