Cirrus CI free usage going away - job runtime issues & credits

[rgommers]

This is getting kinda annoying: [skip cirrus] is broken and the wheel build and other CI jobs are running way too often. It was probably broken by the addition of other logic like always triggering on PRs with build-related label.

[charris]

What do you want to happen with the build label? I use it when I want to test wheel builds. It is a bit annoying, as simply adding a label will retrigger the builds.

[rgommers]

Labels are for indicating what a PR or issue is about, so I add it to build-related PRs. And running wheel builds by default is fine, but the [skip cirrus] should supercede that (just like it does for [skip actions] on GHA).

[charris]

but the [skip cirrus] should supercede

Fair enough, but some labels do trigger wheel builds, so they aren't only about what the PR or issue is about. Not sure how we could change that unless we want to add a [build wheels] option.

[rgommers]

Why [skip cirrus] doesn't work is explained by

numpy/.cirrus.star

Lines 23 to 25 in 7fc7277

	# Obtain commit message for the event. Unfortunately CIRRUS_CHANGE_MESSAGE
	# only contains the actual commit message on a non-PR trigger event.
	# For a PR event it contains the PR title and description.

It looks like it's not the build label though, we run wheel builds on Cirrus always. E.g. look at this doc-only PR with no labels: gh-24277. This wastes a huge amount of compute time.

[charris]

Cirrus is a bit different. One problem is that it cannot be manually run, I think it needs something in the *.yml file for that. I also think (only) wheels are built and tested, I'm not sure how a build can be made without that. It is on my problem list, but not annoying enough to spend time on it. Maybe @andyfaff has some thoughts.

[andyfaff]

.cirrus.star is triggered to run with a lot of GH events on the numpy/numpy repo, e.g. if there are commits to PRs, commits to branches, PRs are opened, labels are attached to PRs, tags are pushed, merges, etc.

Here is the current logic for numpy's cirrus CI, as contained in .cirrus.star:

• only run jobs only on the numpy/numpy repo.
• (do a wheels build if a 'nightly' cron job is requested)
• for all other triggers (and there can be a lot of them) the configuration script looks at the commit message of the SHA, provided by CIRRUS_CHANGE_IN_REPO, for the event. If there is [skip cirrus] or [skip ci] in the message then don't run any CI.
• otherwise do the wheel build and the macosx_arm64 runs.

This logic will dictate that the wheels always get built if there is no [skip cirrus] in the commit message.

[skip cirrus] is broken and the wheel build and other CI jobs are running way too often.

I'm pretty sure [skip cirrus] works for commits to PRs. But for other events if the commit message for the SHA doesn't contain those words then the wheel build and macosx_arm64 will run. For example adding labels to a PR will trigger these runs if the last commit to the PR doesn't contain the magic words.

It's not clear to me from previous comments what additional logic is being requested. The following environment variables may be useful in reducing the number of runs made:

• CIRRUS_PR | PR number if current build was triggered by a PR - we may be able to query Github as to what occurred in the PR to trigger cirrus-ci. We already query GH in cirrus.star.
• CIRRUS_PR_LABELS | comma separated list of PR's labels if current build was triggered by a PR - examine the labels to see if 36 - build is present. e.g. if you look in the debugging info for https://cirrus-ci.com/build/4758553810960384 you'll see that label present.
• CIRRUS_TAG | Tag name if current build was triggered by a new tag.

Possible extra logic that could be done:

• don't automatically do wheel build unless ...
• it's a nightly job
• scan SHA message for [wheel build], do wheel build if present
• scan CIRRUS_PR_LABELS for 36 - Build, do wheel build if present
• build wheels for a tag event

Relevant links:

CIRRUS_CHANGE_IN_REPO
numpy's .cirrus.star config
numpy's wheel build config
numpy's macosx_arm64 config
cirrus environment variables

EDIT:
[skip cirrus] wasn't working, fixed in #24285.

[andyfaff]

Having just said that I see an issue at https://github.com/numpy/numpy/blob/main/.cirrus.star#L27. I'll open a PR.

[andyfaff]

You can examine what we requested from the GH API in 24282. The github request is:
https://api.github.com/repos/numpy/numpy/git/commits/cad8595a8c86c173285d82b61f6797ff24324364.

This is what is returned:

{
 "sha": "cad8595a8c86c173285d82b61f6797ff24324364",
 "node_id": "C_kwDOAA3dP9oAKGNhZDg1OTVhOGM4NmMxNzMyODVkODJiNjFmNjc5N2ZmMjQzMjQzNjQ",
 "url": "https://api.github.com/repos/numpy/numpy/git/commits/cad8595a8c86c173285d82b61f6797ff24324364",
 "html_url": "https://github.com/numpy/numpy/commit/cad8595a8c86c173285d82b61f6797ff24324364",
 "author": {
 "name": "Andrew Nelson",
 "email": "andyfaff@gmail.com",
 "date": "2023-07-29T00:15:13Z"
 },
 "committer": {
 "name": "Andrew Nelson",
 "email": "andyfaff@gmail.com",
 "date": "2023-07-29T00:15:13Z"
 },
 "tree": {
 "sha": "1c9f90fbbcff3162542b6663e0fe75a86e819bb4",
 "url": "https://api.github.com/repos/numpy/numpy/git/trees/1c9f90fbbcff3162542b6663e0fe75a86e819bb4"
 },
 "message": "CI: correct URL in cirrus.star [skip cirrus]",
 "parents": [
 {
 "sha": "422854fa8dc501e5fcbd713093fdee04e7e9ebb8",
 "url": "https://api.github.com/repos/numpy/numpy/git/commits/422854fa8dc501e5fcbd713093fdee04e7e9ebb8",
 "html_url": "https://github.com/numpy/numpy/commit/422854fa8dc501e5fcbd713093fdee04e7e9ebb8"
 }
 ],
 "verification": {
 "verified": false,
 "reason": "unsigned",
 "signature": null,
 "payload": null
 }
}

The cirrus CI didn't run because [skip cirrus] is in dct['message'].

[andyfaff]

I think we should be able to limit the wheel build to whatever is in the GHA wheels build, if that's desired.

EDIT: apart from manual trigger, not sure how to do that.

[rgommers]

Thanks for the fix @andyfaff!

Given the major reduction in free resources available per 1 Sep (see https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/), I think we have a lot more work to do here unfortunately (and may consider buying some credits).

Regarding the label-based trigger, I think there are two things wrong with it:

• the Build label is wrong for this, it should be a dedicated and clearly named label like trigger-cirrus
• it's a problem that this label, once added to a PR, tends to stay on it and then the wheel builds run for every subsequent push. typically what's intended is a one-off "check wheel builds". It doesn't really make sense (resource-usage wise) to run the full battery of wheel builds on every push to a PR.

Given the above and that our resource usage at the current rate (see screenshot below) is completely unsustainable and would run at ~$2,500/month (or ~$1,400 after the upcoming price reductions also announced in the blog post linked above) if we'd have to pay for it from 1 Sep, I'd much prefer to get rid of label-based triggering completely. Manual wheel build triggers should be rare and reserved to maintainers who know what they are doing and are able to push an empty commit with the correct commands in the commit message.

CPU usage is also bad on cibuildwheel jobs (and note that credits go per CPU-minute, i.e. per core rather than per job); we need to ensure to use 2 cores for pytest:

I'll note that on jobs with 2 CPUs, using -n auto also isn't great, since pytest-xdist translates that to -n4 rather than -n2 and its scaling of parallelism is terrible and performance improvement tends to get negative from 4 jobs already. -n2 on 2-core jobs is already far from linear, 3 is getting questionable but still gains typically, >=4 quickly decreases performance.

Example log from a recent macos_arm64_test run showing we get 4 pytest-xdist workers:

$ /Users/admin/numpy-dev/bin/python3.10 -m pytest --rootdir=/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build-install/usr/lib/python3.10/site-packages -n auto -m 'not slow' numpy
============================= test session starts ==============================
platform darwin -- Python 3.10.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build-install/usr/lib/python3.10/site-packages
configfile: ../../../../../pytest.ini
plugins: hypothesis-6.82.0, xdist-3.3.1
created: 4/4 workers
4 workers [34198 items]

Here is the full list of jobs and runtimes for a single run:

That's a total of ~222 CPU minutes for wheel builds per run, divided in

• 77.5 * 2 = 144 min on Linux aarch64 = $0.145 per run
• 33.5 * 4 = 134 min on macOS = $0.67 per run

So each wheel build costs about $0.80 each time it's triggered - this is a lot. We also have issues with some tests in the full test suite that need fixing (e.g., the slow typing tests shouldn't be run by default, they're the same on all platforms and take well over a minute). But most importantly, we should not be triggering wheel builds so much, they're only very rarely useful.

[rgommers]

As you can see in gh-24289, that PR - which only tweaked a code comment in a meson.build file and I added the 04 - Documentation label to - still triggers a full set of wheel builds on Cirrus (I cancelled them manually after they started running).

[rgommers]

And then after a merge to main, it's running yet again: https://cirrus-ci.com/build/5871472732798976.