Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
It’s a monitoring service SDK. https://sentry.io/for/python/
So the basic unit of parallelism in GitHub Actions is the job. Handily you can define many jobs in a single workflow. For example, pytest-twisted has a matrix with axes for OS, Python version, and the type of Twisted reactor to use. This let's you handle this in a single workflow file instead of one for every environment and also by using the matrix you don't have to repeat all your step definitions even within the single workflow file. https://github.com/pytest-dev/pytest-twisted/blob/56991132000791cf89283ac5aa033f3469c63e35/.github/workflows/ci.yml That is manual, though not terribly painful. So on to generation. Before I started at my current job they had a setup that sounds fairly similar to what was described in the blog post, albeit without being based around tox. It's still not based around tox but I did move the generation aspect into the workflow. This does cost a few seconds delay of each run but also saves the developers from having to remember, or be reminded to, regenerate the workflows and also avoids the repetitive spammy diffs when all the workflow files are updated or new environments are added (or removed). Note that the use of a reusable workflow `test-single.yml` from `test.yml` is unrelated as it is compensating for the 256 job-per-matrix limit and it handles the OS axis of the "matrix". Note it if you need it, but don't get too distracted by it. https://github.com/Chia-Network/chia-blockchain/pull/11722/files#diff-faff1af3d8ff408964a57b2e475f69a6b7c7b71c9978cccc8f471798caac2c88R21 ``` configure: name: Configure matrix runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v3 - name: Setup Python environment uses: actions/setup-python@v2 with: python-version: '3.9' - name: Generate matrix configuration id: configure run: | python tests/build-job-matrix.py --per directory --verbose > matrix.json cat matrix.json echo ::set-output name=configuration::$(cat matrix.json) echo ::set-output name=steps::$(cat some.json) outputs: configuration: ${{ steps.configure.outputs.configuration }} ``` The original runs from the PR and it's merge are gone so this data is from a recent run which presumably has had some non-interesting source changes. Anyways, the point is that the configure job runs a script to generate some json that can be used to generate a list of separate matrix entries we want. The primary factor here is the `"test_files"` key. That's the list of files for pytest to run in each given job. In our case we are breaking it down by test file directory in addition to the platforms and Python versions. Here's just a snippet of the script output, but there's a list of several of these. 34 at present. ``` 2022-11-23T04:53:14.4930893Z 2022-11-23 04:53:14,486: { 2022-11-23T04:53:14.4931712Z 2022-11-23 04:53:14,486: "check_resource_usage": false, 2022-11-23T04:53:14.4934045Z 2022-11-23 04:53:14,486: "checkout_blocks_and_plots": true, 2022-11-23T04:53:14.4935264Z 2022-11-23 04:53:14,486: "enable_pytest_monitor": "", 2022-11-23T04:53:14.4936372Z 2022-11-23 04:53:14,486: "install_timelord": false, 2022-11-23T04:53:14.4937734Z 2022-11-23 04:53:14,486: "job_timeout": 60, 2022-11-23T04:53:14.4938848Z 2022-11-23 04:53:14,486: "name": "blockchain", 2022-11-23T04:53:14.4940448Z 2022-11-23 04:53:14,486: "pytest_parallel_args": { 2022-11-23T04:53:14.4941525Z 2022-11-23 04:53:14,486: "macos": " -n 4", 2022-11-23T04:53:14.4942710Z 2022-11-23 04:53:14,486: "ubuntu": " -n 4", 2022-11-23T04:53:14.4943864Z 2022-11-23 04:53:14,486: "windows": " -n 2" 2022-11-23T04:53:14.4944694Z 2022-11-23 04:53:14,486: }, 2022-11-23T04:53:14.4946837Z 2022-11-23 04:53:14,486: "test_files": "tests/blockchain/test_blockchain.py tests/blockchain/test_blockchain_transactions.py" 2022-11-23T04:53:14.4948036Z 2022-11-23 04:53:14,486: }, ``` This is used in the test job matrix as `${{ fromJson(inputs.configuration) }}`. https://github.com/Chia-Network/chia-blockchain/pull/11722/files#diff-2eb322797408fca9558916ce3aee3c2d2a8f1dada7499a63bdbc3bacc4b45559R39 ``` jobs: test: name: ${{ matrix.os.emoji }} ${{ matrix.configuration.name }} - ${{ matrix.python.name }} runs-on: ${{ matrix.os.runs-on }} timeout-minutes: ${{ matrix.configuration.job_timeout }} strategy: fail-fast: false matrix: configuration: ${{ fromJson(inputs.configuration) }} ``` There are also `os:` and `python:` matrix axes, but this `configuration:` line is the interesting bit creating the 34x configurations in that matrix axis. I first started doing this generative matrix pattern in https://github.com/altendky/romp where I wanted to be able to trigger a completely generic CI definition (in Azure Pipelines in this case) from the CLI and provide it an arbitrary matrix definition. To be clear, when I did that I did find a few other folks that had done this so I am not trying to claim credit for the idea. Also note that there's https://pypi.org/project/pytest-xdist/. Depending on your setup it may be more worthwhile to do the in-runner concurrency with tox, but at least be aware of the pytest-level option, especially for local runs with more cores. And yes, as with most of these things there are some nasty corners you may or may not run into when using pytest-xdist. But, it can also speed things up significantly even with just two cores. So, keep going, maybe save the devs from both code generation and committing generated code. And, enjoy the matrix...
So the basic unit of parallelism in GitHub Actions is the job. Handily you can define many jobs in a single workflow. For example, pytest-twisted has a matrix with axes for OS, Python version, and the type of Twisted reactor to use. This let's you handle this in a single workflow file instead of one for every environment and also by using the matrix you don't have to repeat all your step definitions even within the single workflow file. https://github.com/pytest-dev/pytest-twisted/blob/56991132000791cf89283ac5aa033f3469c63e35/.github/workflows/ci.yml That is manual, though not terribly painful. So on to generation. Before I started at my current job they had a setup that sounds fairly similar to what was described in the blog post, albeit without being based around tox. It's still not based around tox but I did move the generation aspect into the workflow. This does cost a few seconds delay of each run but also saves the developers from having to remember, or be reminded to, regenerate the workflows and also avoids the repetitive spammy diffs when all the workflow files are updated or new environments are added (or removed). Note that the use of a reusable workflow `test-single.yml` from `test.yml` is unrelated as it is compensating for the 256 job-per-matrix limit and it handles the OS axis of the "matrix". Note it if you need it, but don't get too distracted by it. https://github.com/Chia-Network/chia-blockchain/pull/11722/files#diff-faff1af3d8ff408964a57b2e475f69a6b7c7b71c9978cccc8f471798caac2c88R21 ``` configure: name: Configure matrix runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v3 - name: Setup Python environment uses: actions/setup-python@v2 with: python-version: '3.9' - name: Generate matrix configuration id: configure run: | python tests/build-job-matrix.py --per directory --verbose > matrix.json cat matrix.json echo ::set-output name=configuration::$(cat matrix.json) echo ::set-output name=steps::$(cat some.json) outputs: configuration: ${{ steps.configure.outputs.configuration }} ``` The original runs from the PR and it's merge are gone so this data is from a recent run which presumably has had some non-interesting source changes. Anyways, the point is that the configure job runs a script to generate some json that can be used to generate a list of separate matrix entries we want. The primary factor here is the `"test_files"` key. That's the list of files for pytest to run in each given job. In our case we are breaking it down by test file directory in addition to the platforms and Python versions. Here's just a snippet of the script output, but there's a list of several of these. 34 at present. ``` 2022-11-23T04:53:14.4930893Z 2022-11-23 04:53:14,486: { 2022-11-23T04:53:14.4931712Z 2022-11-23 04:53:14,486: "check_resource_usage": false, 2022-11-23T04:53:14.4934045Z 2022-11-23 04:53:14,486: "checkout_blocks_and_plots": true, 2022-11-23T04:53:14.4935264Z 2022-11-23 04:53:14,486: "enable_pytest_monitor": "", 2022-11-23T04:53:14.4936372Z 2022-11-23 04:53:14,486: "install_timelord": false, 2022-11-23T04:53:14.4937734Z 2022-11-23 04:53:14,486: "job_timeout": 60, 2022-11-23T04:53:14.4938848Z 2022-11-23 04:53:14,486: "name": "blockchain", 2022-11-23T04:53:14.4940448Z 2022-11-23 04:53:14,486: "pytest_parallel_args": { 2022-11-23T04:53:14.4941525Z 2022-11-23 04:53:14,486: "macos": " -n 4", 2022-11-23T04:53:14.4942710Z 2022-11-23 04:53:14,486: "ubuntu": " -n 4", 2022-11-23T04:53:14.4943864Z 2022-11-23 04:53:14,486: "windows": " -n 2" 2022-11-23T04:53:14.4944694Z 2022-11-23 04:53:14,486: }, 2022-11-23T04:53:14.4946837Z 2022-11-23 04:53:14,486: "test_files": "tests/blockchain/test_blockchain.py tests/blockchain/test_blockchain_transactions.py" 2022-11-23T04:53:14.4948036Z 2022-11-23 04:53:14,486: }, ``` This is used in the test job matrix as `${{ fromJson(inputs.configuration) }}`. https://github.com/Chia-Network/chia-blockchain/pull/11722/files#diff-2eb322797408fca9558916ce3aee3c2d2a8f1dada7499a63bdbc3bacc4b45559R39 ``` jobs: test: name: ${{ matrix.os.emoji }} ${{ matrix.configuration.name }} - ${{ matrix.python.name }} runs-on: ${{ matrix.os.runs-on }} timeout-minutes: ${{ matrix.configuration.job_timeout }} strategy: fail-fast: false matrix: configuration: ${{ fromJson(inputs.configuration) }} ``` There are also `os:` and `python:` matrix axes, but this `configuration:` line is the interesting bit creating the 34x configurations in that matrix axis. I first started doing this generative matrix pattern in https://github.com/altendky/romp where I wanted to be able to trigger a completely generic CI definition (in Azure Pipelines in this case) from the CLI and provide it an arbitrary matrix definition. To be clear, when I did that I did find a few other folks that had done this so I am not trying to claim credit for the idea. Also note that there's https://pypi.org/project/pytest-xdist/. Depending on your setup it may be more worthwhile to do the in-runner concurrency with tox, but at least be aware of the pytest-level option, especially for local runs with more cores. And yes, as with most of these things there are some nasty corners you may or may not run into when using pytest-xdist. But, it can also speed things up significantly even with just two cores. So, keep going, maybe save the devs from both code generation and committing generated code. And, enjoy the matrix...