kube-monkey VS kops

Not in the typical sense, but we have plenty of standard practices and cross-team checkpoints to limit risk. By the time we're deploying changes to production, the work has had a card created, assigned points (which necessarily involves discussing scope and risk), architected (as a group), code peer reviewed, hit unit tests (automated), integration tests (automated), functional tests (automated), smoke tested (automated) end-to-end tests (a few automated, but mostly manual by QA), acceptance tested (by QA and business), resilience tests (chaos engineering with kube-monkey), been deployed to at least 3 environments (with the same exact same artifacts, just with config changes), and monitored for failures (pod restarts, log anomalies, etc -- all automated). Deploy to production is well communicated, and ANY team can request a halt to the deploy if they have concerns.

Part of being a cloud native company means designing services for failure. What happens, for example, if the payment service/pod goes down? Do the rest of your services continue operating normally? One thing tools like kube-monkey does is automatically kill pods for you on a certain date at a certain time in order to plan for failure events. Just wondering if anyone has dove into the deep end with this type of tooling and really just gone all out, besides Netflix?

Chaos Mesh is a popular solution (about 5k GitHub stars), but — obviously — not the only one. E.g., Litmus is a powerful platform to test many things, and kube-monkey might be a good option for more basic stuff.

Exactly the kind of thing I love to see! Sounds like a great use case for a tool like kube-monkey as well.

Kube Monkey

The general problem of patching resource definitions that are not fully under your control has also been recognized for some time. This is true of default resources created and updated by cluster maintenance tools (e.g. kOps), or by public helm charts that you use to install common services and operators (e.g. nginx-ingress, cert-manager, and so on). High quality charts will allow you to override the configuration of important components such as service account references, but some simpler charts offer much less configuration.

The kops documentation is not very clear about this.

Very fast progress and great article with lots of useful information. I found myself nodding a lot having recently seen GitLab Dedicated https://docs.gitlab.com/ee/subscriptions/gitlab_dedicated/ being developed that had my similar challenges. I wonder what other people think of Kops https://kops.sigs.k8s.io/

note: I work with kube on a daily basis. All current clusters under my care use a managed backplane service though there are a number of small (3+ years-old) clusters that I still keep an eye on that were initialized using kops.

kOps with your own instances.

Companies run their own clusters (sometimes for cost reasons), using tools like kops, and kubeadm to set up their own clusters.

curl -Lo kops https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64 chmod +x kops sudo mv kops /usr/local/bin/

Kops is a cluster setup and management command line tool that deploys a Kubernetes cluster to AWS. It provides configuration abstractions such as manifest YAML files that facilitate node and components configuration. And like Ansible, it will provide dry-run capabilities and ensures idempotency of changing the nodes.