gitlab-ci-local VS cache

Among these options, the one that has gained the most traction is gitlab-ci-local :

The main reason behind this change is to have consistent scripts for local testing and remote runners during testing and debugging. However, there are already tools available, such as gitlab-ci-local, that allow you to run jobs locally, partially invalidating this argument. Additionally, working locally may not provide access to all necessary variables.

Two tools I've used for local Gitlab CI runs: - https://github.com/firecow/gitlab-ci-local - https://gitlab.com/AdrianDC/gitlabci-local

What you might want to look at is this, to meet both needs https://github.com/firecow/gitlab-ci-local

https://github.com/firecow/gitlab-ci-local exists but its not quite there yet. Personally Ive resorted to setting up a self-managed instance at home, relying on the included linter/validator and pushing repeatedly as before.

If you already have/had a working pipeline then maybe https://github.com/firecow/gitlab-ci-local has something worth looking at.

> One thing that would be nice, however, would be the ability to run the entire pipeline locally.

This cost me many hours of waiting for the Gitlab CI runner when debugging non-trivial pipelines, when the issue was something that did not have to do with the script steps inside of the jobs but rather how the Gitlab runner handled things.

I've found gitlab-ci-local [1] which actually does run the Gitlab pipeline locally, although I had to write some boilerplaye scripts to set up all the necessary 'CI_FOO_SOMETHING' environment variables before running the tool. (Which sometimes came back to bite me because the issue was actually in the content of some of those environment variables). It's still a very good tool.

[1] https://github.com/firecow/gitlab-ci-local

Most CI/CD tools let you run pipelines locally. Just one example: https://circleci.com/blog/using-runner-for-local-testing/ . In my opinion Gitlab and Circleci have the test tools for this.

> with no persistent storage

There's https://github.com/actions/cache though?

Use Cache Actions: GitHub Actions provides cache actions that simplify caching implementation. The @actions/cache JavaScript library is a popular choice for managing caching in workflows. It offers flexible options for storing and retrieving cache artifacts based on keys, scopes, and paths.

GitHub maintain a set of repos called actions. One of which is called cache.

By utilizing the actions/cache action action, we implemented a strategy to store and retrieve dependencies, preventing redundant installations.

I do think it's good practice to enable caching, such that your script doesn't hit RubyGems / pip / npm / etc every time it runs.

That way at least the automation activity stays entirely within the GitHub / Azure network.

It looks like you can do that for Ruby by adding this:

https://github.com/actions/cache/blob/master/examples.md#rub...

    - uses: ruby/setup-ruby@v1

➜ getting-started-with-act git:(master) act -j build WARN ⚠ You are using Apple M1 chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠ [Node.js CI/build] 🚀 Start image=node:16-buster-slim [Node.js CI/build] 🐳 docker pull image=node:16-buster-slim platform= username= forcePull=false [Node.js CI/build] 🐳 docker create image=node:16-buster-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] [Node.js CI/build] 🐳 docker run image=node:16-buster-slim platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] [Node.js CI/build] ☁ git clone 'https://github.com/actions/setup-node' # ref=v3 [Node.js CI/build] ☁ git clone 'https://github.com/actions/cache' # ref=v3 [Node.js CI/build] ☁ git clone 'https://github.com/actions/upload-artifact' # ref=v3 [Node.js CI/build] ⭐ Run Main actions/checkout@v3 [Node.js CI/build] 🐳 docker cp src=/Users/andrewevans/Documents/projects/getting-started-with-act/. dst=/Users/andrewevans/Documents/projects/getting-started-with-act [Node.js CI/build] ✅ Success - Main actions/checkout@v3 [Node.js CI/build] ⭐ Run Main Use Node.js 16.x [Node.js CI/build] 🐳 docker cp src=/Users/andrewevans/.cache/act/actions-setup-node@v3/ dst=/var/run/act/actions/actions-setup-node@v3/ [Node.js CI/build] 🐳 docker exec cmd=[node /var/run/act/actions/actions-setup-node@v3/dist/setup/index.js] user= workdir= [Node.js CI/build] 💬 ::debug::isExplicit: [Node.js CI/build] 💬 ::debug::explicit? false

You could try to find some point in the process where you can set up Actions caches with actions/cache, otherwise Container customization for Self-Hosted Runners is currently in Beta.

I would configure Github Actions cache so Docker doesn't have to compile all layers from scratch every time

In the copied example, npm caching is done via actions/cache@v2 action. But we can simplify our workflow by dropping this step and using built-in functionality for caching