kubebuilder VS ginkgo

The Spin operator uses the Kubebuilder framework and contains a Spin App Custom Resource Definition (CRD) and controller. It watches Spin App Custom Resources and realizes the desired state in the K8s cluster. Aside from the immediate benefits gained by running Wasm workloads in k8s, additional optimizations such as Horizontal Pod Scaling (HPA) and k8s Event-driven Autoscaling (KEDA) can be achieved in a pinch.

kubebuilder: brew install kubebuilder

The operator that I've been working on is designed to manage the full lifecycle of a QuestDB database instance, including version and hardware upgrades, config changes, backups, and (eventually) recovery from node failure. I used the Operator SDK and kubebuilder frameworks to provide scaffolding and API support.

Recently, I've been spending a lot of time writing a Kubernetes operator using the go operator-sdk, which is built on top of the Kubebuilder framework. This is a list of a few tips and tricks that I've compiled over the past few months working with these frameworks.

Since we built our operator using the Kubebuilder framework, most standard monitoring tasks were handled for us out-of-the-box. Our operator automatically exposes a rich set of Prometheus metrics that measure reconciliation performance, the number of k8s API calls, workqueue statistics, and memory-related metrics. We we were able to ingest these metrics into pre-built dashboards by leveraging the grafana/v1-alpha plugin, which scaffolds two Grafana dashboards to monitor Operator resource usage and performance. All we had to do was add these to our existing Grafana manifests and we were good to go!

I wrote a CSI driver and some operators. I admire K8s, because you can find solution to almost any problem in the source code - API versioning, load balancing, request throttling, optimistic concurrency, security, and much much more. I recommend https://book.kubebuilder.io/ It is similar to Operator SDK, but without Openshift-specific stuff. It gradually introduces you to many k8s concepts, and follows design patterns that k8s uses internally.

Do you mean this: https://book.kubebuilder.io/ ?

https://book.kubebuilder.io/ all you need to know

A better way to write an operator these days is to use kubebuilder [1].

My complaint is that I have seen orgs write operators for random stuff, often reinventing the wheel. Lot of operators in orgs are result of resume driven development. Having said that it often comes handy for complex orchestration.

[1]https://github.com/kubernetes-sigs/kubebuilder

Ginkgo: a testing framework based on the concept of ‌"Behavior Driven Development" (BDD)

We were also able to leverage Ginkgo's parallel testing runtime to run our integration tests on multiple concurrent processes. This provided multiple benefits: we could run our entire integration test suite in under 10 minutes and also reuse the same suite to load test the operator in a production-like environment. Using these tests, we were able to identify hot spots in the code that needed further optimization and experimented with ways to save API calls to ease the load on our own Kubernetes API server while also staying under various AWS rate limits. It was only after running these tests over and over again that I felt confident enough to deploy the operator to our dev and prod clusters.

A bit off-topic, but i really like the ginkgo BDD framework

You obviously are not familiar with the third circle of golang continuous integration hell that is ginkgo+gomega:

https://onsi.github.io/ginkgo/#adding-specs-to-a-suite

It’s actually worse than that example suggests. Stuff like Expect(“type safety”).ShouldBe(GreaterThan(13)) throws runtime errors.

The semantics of parallel test runs weren’t defined anywhere the last time I checked.

Anyway, you’ll be thinking back fondly to the days of TestShouldReplaceChildrenWhenUpdatingInstance because now you need to write nested function calls like:

Context(“instances”, func …)

Describe(“that are being updated”, …)

Expect(“should replace children”, …)

And to invoke that from the command line, you need to write a regex against whatever undocumented and unprinted string it internally concatenates together to uniquely describe the test.

Also, they dump color codes to stdout without checking that they are writing to a terminal, so there will be line noise all over whatever automated test logs you produce, or if you pipe stdout to a file.

If you are using Ginkgo for your e2e, this library might of help.

https://onsi.github.io/ginkgo/ offers a simple way to create tables with different scenarios useful to generate different test cases based on a file like a yml without to need to develop useless code. Maybe at start seems to be a little verbose but depends how you design the test case.

What are people’s thoughts on testing frameworks? I’ve heard that most devs only use the testing package in the standard library and the testify package for assertions— I assume this is because Go is meant to be lightweight and scalable, and adding external dependencies basically goes against that. But I’ve also seen devs use packages like ginkgo to make tests more structured and readable. What do you guys think?

Ginkgo Behavioural test framework

If you write your tests with Ginkgo [0] its CLI can do this for you. It also has nice facilities to quickly disable a test or portion of a test by pretending an X to the test function name, or to focus a test (only run that test) by prepending an F. It’s pretty nice.

[0]: https://onsi.github.io/ginkgo/

The BDD testing framework Ginko [1] has some "weird" / unidiomatic patterns, yet it is very popular

https://github.com/onsi/ginkgo