ci: add rust-cache action

[Firestar99]

Requires #530

• implement rust-cache in our CI
• "base" cache
  • "test" job creates the cache
  • "compiletest" and "difftest" just use the cache
  • "test" warms the cache for "compiletests" and "difftests", saves ~2min each
• "android" cache, only for the android action since it compiles for an arm target
• lint get's no cache, while speedup is significant (6m -> 1m), not worth the cache space cost
• github has a cache usage limit of 10GB, this is using ~4GB per branch
• thus, caches are only created on main, but can be reused on branches based on main

(slightly outdated table, should still be accurate enough)

job	windows	mac	linux	total
test	620 MB	580 MB	730 MB	1930 MB
difftest	230 MB	240 MB	240 MB	710 MB
android			460 MB	460 MB
vulkan-sdk	290 MB		460 MB	750 MB
total	1140 MB	820 MB	1890 MB	3850 MB

Final Results

• Results (same table, new values for "test cache"):
  • own cache @ de88135 (every job it's own cache, incremental build, without ci: free speedups #530)
  • test cache @ 7519e71 (incremental build with good cache, cherry-picked ci: free speedups #530)
  • test clean @ 7519e71 (clean build with cache creation, cherry-picked ci: free speedups #530)
  • main @ 6441dc7 (without ci: free speedups #530)

job	windows	mac	linux
test own cache	9m 26s	11m 54s	4m 59s
test test cache	10m 31s	6m 17s	5m 37s
test test clean	27m 48s*	20m 25s	15m 15s
test main	18m 10s	15m 41s	11m 28s
compiletest own cache	14m 17s	7m 43s	6m 12s
compiletest test cache	14m 40s	8m 42s	6m 21s
compiletest test clean	16m 38s*	11m 45s	8m 32s
compiletest main	19m 32s	14m 42s	8m 3s
difftest own cache	15m 15s	10m 29s	8m 22s
difftest test cache	15m 13s	9m 13s	7m 44s
difftest test clean	23m 43s	15m 40s	11m 51s
difftest main	26m 30s	15m 0s	14m 30s
lint own cache			1m 42s
lint test cache			5m 21s
lint test clean			6m 14s
lint main			5m 58s

End to end:

• own cache: 15m 41s
• test cache: 15m 24s
• test clean: 27m 59s*
• test main: 26m 41s

*adjusted due to flakyness of rustup install, usually takes only 40s on windows

Analysis

1. Speedup with a good cache, before ci: free speedups #530? negative = speedup, "main" - "test cache"

job	windows	mac	linux
test	-8m 45s	-4m	-6m 30s
compiletest	-5m	-7m	-2m
difftest	-11m 15s	-4m 30s	-6m

End to end: -11m 20s

2. Overhead of a clean build without caches? positive = slowdown, "test clean" - "main"

Note: this includes #530

job	windows	mac	linux
test	+9m 45s	+5m	+4m
compiletest	-3m	-3m	-30s
difftest	-3m	-40s	-2m 30s

End to end: +1m 20s*

3. How much faster would individual caches vs current shared "test" cache be? negative = speedup, "test cache" - "own cache"

Note: This is unfair as "own cache" data does not include #530. Expect difftest to be worse and test to be better than a retest with that PR.

job	windows	mac	linux
test	-1m	flaky	-40s
compiletest	-20s	-1m	9s
difftest	+2s	+1m 10s	+40s

End to end: +20s

My investigation log

Feel free to ignore and just look at the results

Adding caching

First run (with no cache present) is looking real bad:

• windows test +1m 8s
• windows difftest +6m 24s
  • no idea why the cache would take so long when there's nothing to download and nothing has been compiled???
• all "install rust toolchain" are now 0s, so we probably have to move them before the cache action

First CI with restore

Comparing latest main to this PR:

• overall time: 27m 39s -> 16m 8s
• longest job "windows-difftest": 27m 29s -> 16m 0s
• cache restore, measured from seconded empty commit in this PR, which may have been removed:
  • windows: 58s to 2m 3s
    • first run of windows difftest has an obscene "restore" of 6m 24s even though there's nothing to restore???
  • linux: ~30s
  • mac: 30s - 40s
• cache creation, measured from first commit in this PR:
  • windows: 30s - 1m
  • linux: 8s - 15s
  • mac: 30s - 50s

Install toolchain before restore (run attempt #1)

• rust-cache seems to install the toolchain for you via rustup if you don't do it yourself, toolchain install has 0s execution time
• Action: move it before cache restore
• Results: old rust-cache time is now split between rustup install and new rust-cache
• Could caches contain rustup builds? Try purging caches

purging caches and retrying CI (run attempt #2)

• as expected, a lot of runtime is moved from "rust-cache" to "install toolchain"
• rust-cache restore times:
  • windows: 2-3s
  • mac: 1-2s
  • linux: 1-2s
  • => couldn't repo crazy CI times from first attempt, just a fluke?
• cache creation:
  • windows: 30-50s
  • mac: 30-40s, 60s fluke?
  • linux: ~10s
• Results: rustup before rust-cache to separate out toolchain install time

Lessen cache usage

• github cache is reporting high cache usage
  • 10GB limit by gh
  • caches take up:

job	windows	mac	linux	total
test	620 MB	590 MB	740 MB	1950 MB
compiletest	310 MB	300 MB	320 MB	930 MB
difftest	550 MB	550 MB	560 MB	1660 MB
lint			720 MB	720 MB
android			460 MB	460 MB
vulkan-sdk	290 MB	500 MB	460 MB	1250 MB
total	1770 MB	1940 MB	3260 MB	6970 MB

• Idea: can we reuse the same cache between actions?
  • only test creates the cache, for each platform
  • difftest may get a separate cache for the difftest target dir
• attempt #1, creating caches
  • windows randomly taking 15min to rustup install toolchain, likely the culprit of rust-cache taking so long the first time?
• attempt #2, restoring caches
  • Mac: vulkan sdk caching doesn't work at all, but still creates new caches? Cache restore on MacOS broken jakoch/install-vulkan-sdk-action#558
  • jobs have different cache hash due to rustup env vars... retry entire experiment

remove rustup env vars

• cache usage (limit of 10GB):

job	windows	mac	linux	total
test	620 MB	580 MB	730 MB	1930 MB
difftest	230 MB	240 MB	240 MB	710 MB
android			460 MB	460 MB
vulkan-sdk	290 MB	500 MB	460 MB	1250 MB
total	1140 MB	1320 MB	1890 MB	4350 MB

• CI times:
  • own cache @ de88135
  • test cache @ 7519e71
  • main @ 79acad1 (without ci: free speedups #530)

job	windows	mac	linux
compiletest own cache	14m 17s	7m 43s	6m 12s
compiletest test cache	16m 4s	7m 28s	6m 21s
compiletest main	19m 32s	14m 42s	8m 3s
difftest own cache	15m 15s	10m 29s	8m 22s
difftest test cache	21m 18s	16m 11s	11m 8s
difftest main	26m 30s	15m 0s	14m 30s
lint own cache			1m 42s
lint test cache			6m 1s
lint main			5m 58s

• Analysis of difftest slowdown:
  • windows & mac +6min:
    • +3min from "test difftest" aka. cargo test -p difftest
    • +3min from "difftest" themselves
  • linux follows similar pattern but less worse

move "test difftest" to test job

• Idea: On main windows difftest:
  • test difftest: 7m 29s
  • difftest: 14m 44s
  • move work from the longest running task difftest to test
  • maybe also improve rust-cache reuse, and speed up difftest some more?
• Results (incremental only)
  • difftest-windows: from 21m 18s to 16m 44s
  • test-windows: 9m 46s to 9m 13s, should have increased, likely just run to run variance
  • caching was useless, almost nothing is shared between "test difftest" and "difftest"
  • free -5min win 🎉

ci: make "test" prep better cache for compiletests and difftests

didn't work, try again

ci: fix prep cache being useless due to weird feature selection

• Idea: both compiletest and difftest spent 1-2min compiling the test runners, cache that
• see final results below

[nazar-pc]

In my projects I usually disable caching on Windows. Their gzip or whatever compression they use is so spectacularly slow that it is faster to download/install things from scratch than decompressing/recompressing cache.

[LegNeato]

I feel like we had this somewhere (rust-cuda?) and had to remove it

[Firestar99]

Well, you've actively using it in rust-cuda, even if for windows only... Otherwise I can't find any mention of it, neither in issues, PRs or history of those files

[Firestar99]

Just rebased it and it started failing, no idea why. The diff to main is just the cache actions, and they're not caching anything since no cache is available, and uploading only happens on main.