gg VS Buildkite

Another interesting option if you are feeling adventurous is using AWS lambda as your build executor.  I have no idea how feasible this is, however, the gg project from Stanford looks interesting.  It attempts to use AWS lambdas for running builds at the maximum possible parallelism.

This is very similar to https://github.com/StanfordSNR/gg by Keith Weinstein's group at Stanford. Cool stuff! Have always wondered why Keith's work hasnt made its way into large companies (besides I think Facebook? not sure)

Running it locally will always be faster as long as your machine is not a bottleneck (#cores, ram, ...). I think the use-case for distcc et al is to enable less-powerful machines to run builds faster by levering other machines. That’s exactly what we use it for at work. Our developers have not-so-powerful laptops and with distcc/icecc they can utilize the power of our build agents in the server room.

Also interesting to read: https://github.com/StanfordSNR/gg

buildkite.com you can run agents where ever you want.

Give https://buildkite.com/ for CI/CD a try, you'll love it!

Even that niche has better competitors, I think. I haven't used on-prem CCI, but I used hosted CCI for several years and when my team switched to https://buildkite.com/ it was a huge breath of fresh air. I think BuildKite is the only CI system I've used I thought was actually worth paying for, and I bet it works out cheaper than self-hosted CCI in most cases as well.

Buildkite: a startup that offers a CI/CD platform that combines the power of your own build infrastructure with the convenience of a managed, centralized web UI.

At Redpanda, we want to always provide an experience that is fast, simple, and productive for developers. That applies to our own team of engineers, too. When considering how we could achieve a more stable continuous integration (CI) pipeline, we wanted that same experience: fast, simple, productive. By running multiple instances of our pipeline steps in parallel on our CI platform, Buildkite, we can now run multiple repetitions of the same Buildkite step and use only the amount of time needed for a single step.

> You can also get comparable performance out of https://buildkite.com/ which lets you self-host runners on AWS

you can self-host github runners as well, with a few caveats, the most serious one being that then you are responsible for cleaning up the state of your self-hosted runner between runs

https://docs.github.com/en/actions/hosting-your-own-runners/...

structural isolation guarantees of the form (build execution during run N cannot possibly impact build execution of run N+1) are tremendously helpful, if you cannot offer similar guarantees when self hosting then it may not be wise to self host.

A better option, if you want to run the builds on your own hardware is to look at something like Buildkite or GitLab CI.

One of the integrations that sends Change Events to PagerDuty is Buildkite. Buildkite is a continuous integration pipeline that allows you to notify PagerDuty with just the information you want to know. Conditional rules allow you to send only what matters.