racker VS typhoon

If someone wants to try out IPMI locally with VMs, here is a setup for that: https://github.com/kinvolk/racker/tree/main/racker-sim (you don't need to use Flatcar and Racker)

When working on automated bare-metal provisioning of Kubernetes clusters (the main purpose of the repo https://github.com/kinvolk/racker) it's quite time consuming to test on actual hardware. Also, the hardware may be in use at the moment, or you don't even have access to a free full rack. It was nice to see that there is IPMI support for QEMU and that ipmi_sim integrates, but it was quite hard to set up. Especially the requirement that the BMCs should get their IP address via DHCP on the internal network is a bit more complicated/hacky. If you need to test IPMI commands or develop your own IPMI automation, have a look at this, otherwise it may serve as a demo on how to do Immutable Infra with your on-prem servers.

Don't forget the lies of cost savings that the Cloud providers have shoved down our industry's throats. We are paying out the nose for cloud services and we are giving up all the rights to our data. It's a bad deal in the end.

I have a bunch of friends that work at SaaS companies and their cloud spend for pretty basic deployments is in the many thousands of dollars a month. Most of their deployments could be handled by a half rack with beefy servers in a couple of datacenters for a fraction of the cost. I pay for a full rack myself and it costs me ~$1200 a month for space, power and bandwidth (10Gb pipe with a current 1Gb commit), and my hardware costs for everything in that rack were a one time cost of around $3000. I have 160 GHz of CPU and 141 GiB of memory for my workloads with a few servers that are not yet provisioned into my Nomad cluster.

And before you say well there are costs involved with finding people that have the skills to do that kind of thing and time needed to set all of that up, yes that is true, but our industry has moved from one bucket to another one that is more expensive in the end with a bunch of downsides. I think there is a middle ground where you can use some cloud services and run the important stuff on hardware you own. The tooling to self-host your own stack in a rack of servers you own is light years better than it was 10 years ago and it keeps getting better. Tools like https://nebula.defined.net/docs/ and https://github.com/poseidon/typhoon for example enable you to use whatever providers you want and build a deployment can cost less, gives you more control over your data, while being agile enough to make changes when the team needs something new or different.

I am excited for the next 10 years of progress and I'd expect we are going to see more companies self-hosting their deployments on bare metal.

This is also why managed Kubernetes is a useful thing (EKS, GKE, et al)... but if you still want to do it yourself, maybe look into some Kubernetes distros (like Typhoon (https://typhoon.psdn.io) which I run on my clusters)

Sounds like this may work for you: https://typhoon.psdn.io/

These days I deploy using Typhoon. Production ready k8s in less than 20 minutes. It leverages Terraform and Flatcar Linux. Lovely combo. No need to do any of it "the hard way" unless you want to know about the internals.

I actually use terraform + matchbox to bare metal provision k8s (flatcar linux) using Typhoon.

Wondering what others think? Typhoon