runner-images VS paths-filter

So you want to create a quick-and-dirty GitHub actions that does only one thing and does it well, or glues together several actions, or simply to show off a bit at your next work interview. Here's how you can do it. Let me introduce you to composite GitHub actions, one of the three types that are there (the other are JavaScript GHAs or container-based GHAs) and maybe one of the most widely unknown. However, they have several things going for them. First, they have low latency: no need to download a container or to set up some JS environment). Second, they are relatively easy to set up: they can be self-contained, with everything needed running directly on the description of the GitHub action. Third, you can leverage all the tools installed on the runner like bash, compilers, build tools... or Perl, which can be that and much more. Even being easy, it is even easier if you have a bit of boilerplate you can use directly or adapt to your own purposes. This is what has guided the creation of the template for a composite GitHub action based on Perl. It is quite minimalistic. But let me walk you through what it has got so that you can use it easier

I used the “ubuntu-latest” image which is Debian-based (with apt) and commonly available in the Github Actions workers, so it deploys very fast.

Yeah this is a good option if you'd like something to deploy yourself! You can also build an AMI from GitHub's upstream image definition (https://github.com/actions/runner-images/tree/main/images/ub...) if you'd like it to match what's available in GitHub-hosted Actions.

With Depot, we're moving towards deeper performance optimizations and observability than vanilla GitHub runners - we've integrated the runners with a cache storage cluster for instance, and we're working on deeper integration with the compute platform that we built for distributed container image builds - as well as expanding the types of builds we can process beyond Actions and Docker, for instance.

But different options will be better for different folks, and the `philips-labs` project is good at what it does.

Whoa, there's a lot of stuff in there [1] that gets installed straight from vendors, without pinning content checksums to a value known-good to Github.

I get it, they want to have the latest versions instead of depending on how long Ubuntu (or, worse, Debian) package maintainers take to package stuff into their mainline repositories... but this attack surface is nuts.

[1] https://github.com/actions/runner-images/tree/main/images/ub...

I had a similar experience with ARC (actions-runner-controller).

One of the machines in the fleet failed to sync its clock via NTP. Once a job X got scheduled to it, the runner pod failed authentication due to incorrect clock time, and then the whole ARC system started to behave incorrectly: job X was stuck without runners, until another workflow job Y was created, and then X got run but Y became stuck. There were also other wierd behaviors like this so I eventually rebuilt everything based on VMs and stopped using ARC.

Using VMs also allowed me to support the use of the official runner images [0], which is good for compatibility.

I feel more people would benefit from managed "self-hosted" runners, so I started DimeRun [1] to provide cheaper GHA runners for people who don't have the time/willingness to troubleshoot low-level infra issues.

[0]: https://github.com/actions/runner-images

Reminds me: Still waiting for native ARM support on GitHub Actions https://github.com/actions/runner-images/issues/5631

Used https://github.com/actions/runner-images to get the packages needed for Ubuntu 22.04 As the packer requires a builder, I used "null" builder to set it as localhost ref: https://developer.hashicorp.com/packer/docs/builders/null (It was way difficult to figure it out the 1st time) I had to modify the .pkr.hcl file to pick my provisioners. I could not understand the use of /opt/hostedtoolcache folder (which I did later)

I also set up recently the policy to onl use merge commits on stable branch, as otherwise the path filter^1 in the workflows would not detect correctly which files changed in a PR.

[1] https://github.com/dorny/paths-filter

I truly don't understand why this isn't more widely discussed (I've seen several "GH Actions Gotchas" where this isn't mentioned). Many of the community actions also seem to be designed to run as short jobs to paper around missing features (for ex: https://github.com/dorny/paths-filter ), that end up eating up an enormous amount of your minutes budget.

If that isn’t sufficient, there are a number of third party workflow steps that enable conditional builds with extra flexibility like https://github.com/dorny/paths-filter

You can use paths-filter to give yourself a bunch of conditional outputs to test against for separate jobs.

That's brilliant. dorny/paths-filter looks like it can eliminate my enumerate job, and then I don't have to concern myself with all this data passing between jobs.

There’s an awkward gotcha/incompatibility between “Required status checks” and workflows that get skipped [1], eg due to setting a “paths” property of a push/pull_request workflow trigger [2].

The checks associated with the workflow don’t run and stay in a pending state, preventing the PR from being merged.

The only workaround I’m aware of is to use an action such as paths-filter [3] instead at the job level.

A further, related frustration/limitation - you can _only_ set the “paths” property [2] at the workflow level (i.e. not per-job), so those rules apply to all jobs in the workflow. Given that you can only build a DAG of jobs (ie “needs”) within a single workflow, it makes it quite difficult to do anything non trivial in a monorepo.

[1]: https://docs.github.com/en/repositories/configuring-branches...

[2]: https://docs.github.com/en/actions/using-workflows/workflow-...

[3]: https://github.com/dorny/paths-filter

We are interested in running a linter only against the modified files. Let's say, we take a look at the provided repo, if I update dags/dummy.py I don't want to waste time and resources running the linter against main.py. For this purpose we use Paths Filter GitHub Action, which is very flexible.

In the spirit of the #ActionsHackathon21, you can see I'm taking advantage of the checkout action GitHub provides and the Paths Filter action by dorny to create the desired workflow. I'm also using the Gistblog Action I created for this hackathon which handles managing all the blog posts as Gists. I'd like to explore Composite actions soon to see if I can reduce all of this to a single action making setup even easier.