k8s-config-connector VS nixery

If your company is k8s centric and the developers are most comfortable with k8s, you might want to focus more on something specific to k8s such as helm, or even if you don't get into helm you may want to use config connector in your yaml to manage GCP resources in an IaC compliant method. You can manage k8s resources with terraform, but if your developers are currently comfortable working directly with k8s you are going to see significant pushback getting them to add terraform as a middleware. You probably still want to manage your GKE clusters and VPCs with terraform since you can't really use config connector.

Config Connector [1] is also an option in this space for GCP, it supports many GCP resources and thus far our experience with it has been largely positive.

[1] https://cloud.google.com/config-connector/docs/overview

We use it heavily with GCP's Kubernetes Config Connector to provision architecture. It could similarly be used for Cloud Functions, etc. given a repo URL that GCP can access. GitOps + operator pattern is a pretty powerful mechanism to let k8s continuously seek state towards your ideal. https://cloud.google.com/config-connector/docs/overview

For K8s, we were using Cloud Composer to do it for us but wanted more fine control over CI/CD, so we decided to go with Airflow on K8s. That's all hosted in GKE now and deployed using ArgoCD with helm. This also led down the IaC rabbit hole which has been a ton of fun too. We use the GCP ConfigConnector resources for that which is a little challenging at first, but gets a lot easier as time goes on.

Terraform is 90% of cloud IaC. But there are newer Kubernetes Operators like Config Connector that can create cloud specific resources.

It works particularly well with Google Config Connector as then it's all just manifests.

The ones from https://nixery.dev/

You can build your own with https://nixery.dev/

I built a service for doing this ad-hoc via image names a few years ago and it enjoys some popularity with CI & debugging use-cases: https://nixery.dev/

name is eerily similar to `nixpkgs`, i.e. the monorepo that defines all packages and one of the underlying technologies here. i get the play on buildpacks, but still, as a nix user it makes me do a double take reading the name

this is neat though, and in political terms, the elevator pitch mentions nix itself as an implementation detail in passing. hopefully, if this catches on, it'll function as a non-threatening gateway drug to nix itself, when users inevitably go digging into the weeds

for anyone interested, prior art on the nix container front: https://nixery.dev

Wow, this is excellent! At a previous job, we had been using k8s + knative to spin up containers on demand, and likewise were unhappy with the delays. Spawner seems excellent.

One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large containers (e.g. a Python base image with a few larger packages installed from PyPI)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit.

I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help.

Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.

Nixery – Docker images on the fly with Nix\ (38 comments)