cachemachine VS nixery

> we found the start times with Kubernetes too slow

Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced?

Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking?

The ones from https://nixery.dev/

You can build your own with https://nixery.dev/

I built a service for doing this ad-hoc via image names a few years ago and it enjoys some popularity with CI & debugging use-cases: https://nixery.dev/

name is eerily similar to `nixpkgs`, i.e. the monorepo that defines all packages and one of the underlying technologies here. i get the play on buildpacks, but still, as a nix user it makes me do a double take reading the name

this is neat though, and in political terms, the elevator pitch mentions nix itself as an implementation detail in passing. hopefully, if this catches on, it'll function as a non-threatening gateway drug to nix itself, when users inevitably go digging into the weeds

for anyone interested, prior art on the nix container front: https://nixery.dev

Wow, this is excellent! At a previous job, we had been using k8s + knative to spin up containers on demand, and likewise were unhappy with the delays. Spawner seems excellent.

One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large containers (e.g. a Python base image with a few larger packages installed from PyPI)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit.

I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help.

Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.

Nixery – Docker images on the fly with Nix\ (38 comments)