CI: dynamic jobs by Kobzol · Pull Request #124175 · rust-lang/rust

Kobzol · 2024-04-19T19:51:27Z

This PR modifies our CI workflows to be dynamic. This means that when a GitHub event is generated, we will run a Python script (calculate-job-matrix.py), which decides which CI jobs should be generated. These jobs are defined in src/ci/github-actions/jobs.yml).

This should provide a few benefits:

Once the migration to dynamic jobs is complete, we shouldn't need expand-yaml-anchors anymore.
The job table on PRs (and also the left job column on auto/try builds) should be much cleaner and contain only the jobs that are actually relevant/executed.
It should be much easier to support dynamic try builds, i.e. to run an arbitrary CI job on a try build.

See this Zulip discussion for more context.

r? @ghost

Kobzol · 2024-04-19T21:20:29Z

@bors try

CI: dynamic jobs Opening as a draft to experiment with [dynamic CI jobs](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/job.20matrix.20re-ordered.20PR.20list). r? `@ghost`

bors · 2024-04-19T21:21:39Z

⌛ Trying commit 8b945c3 with merge 5b35171...

bors · 2024-04-20T05:05:17Z

☔ The latest upstream changes (presumably #124177) made this pull request unmergeable. Please resolve the merge conflicts.

emilyalbini

Left a few minor nits, but this otherwise looks great!

Kobzol · 2024-04-23T07:00:22Z

Thanks for the review, tried to fix all remarks in separate commits.

emilyalbini · 2024-04-23T12:23:42Z

@bors r+

bors · 2024-04-23T12:23:45Z

📌 Commit 2632c61 has been approved by pietroalbini

It is now in the queue for this repository.

Kobzol · 2024-04-23T12:24:44Z

@bors rollup=never

(Who knows what this might do in the auto build).

RalfJung · 2024-04-23T21:22:33Z

+ <<: *job-linux-16c
+
+# Jobs that run when you perform a try build (@bors try)
+# These jobs automatically inherit envs.production, to avoid repeating


Shouldn't they inherit envs.try? (And similar for auto below)

Ah, good point, I forgot to update the comment. I'll add it to a follow up PR.

bors · 2024-04-24T06:40:17Z

⌛ Testing commit 2632c61 with merge e7da0fa...

bors · 2024-04-24T08:42:23Z

☀️ Test successful - checks-actions
Approved by: pietroalbini
Pushing e7da0fa to master...

Kobzol · 2024-04-24T09:09:02Z

Wow, it went through. Great.

rust-timer · 2024-04-24T09:58:29Z

Finished benchmarking commit (e7da0fa): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 673.216s -> 673.226s (0.00%)
Artifact size: 315.45 MiB -> 315.42 MiB (-0.01%)

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Apr 19, 2024

Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from b325a04 to 0e9eed1Compare April 19, 2024 19:56

This comment has been minimized.
Sign in to view

Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from 55550c7 to 476d4aeCompare April 19, 2024 20:45

This comment has been minimized.
Sign in to view

Kobzol force-pushed the ci-dynamic-job branch 4 times, most recently from 65141ab to 8b945c3Compare April 19, 2024 21:19

This comment has been minimized.
Sign in to view

Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 97ca4d3 to 8f21998Compare April 19, 2024 21:33

This comment has been minimized.
Sign in to view

Kobzol added 2 commits April 20, 2024 09:06

Move calculate-job-matrix.py to ci/github-actions
9fec43d

Perform PR and try builds dynamically
7a90679

Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 0efaec5 to 545c12fCompare April 20, 2024 07:18

Refactor calculate-job-matrix.py
6f0ff0b

Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from dc3c544 to 4d3e89cCompare April 20, 2024 07:45

rustbot assigned emilyalbini Apr 20, 2024

emilyalbini reviewed Apr 22, 2024
View reviewed changes

Comment threadsrc/ci/github-actions/jobs.yml Outdated
Comment threadsrc/ci/github-actions/calculate-job-matrix.py Outdated
Comment threadsrc/ci/github-actions/jobs.yml Outdated
Comment threadsrc/ci/github-actions/ci.yml

Kobzol added 3 commits April 23, 2024 08:52

Load GitHub context from environment variables
c251abc

Fix image field in auto jobs
2f2d59d

Make separate envs for try and auto builds
162892b

Fix empty job matrix
2632c61

Kobzol force-pushed the ci-dynamic-job branch from e44fd47 to 2632c61Compare April 23, 2024 07:01

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 23, 2024

RalfJung reviewed Apr 23, 2024
View reviewed changes

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 24, 2024

bors merged commit e7da0fa into rust-lang:masterApr 24, 2024

rustbot added this to the 1.79.0 milestone Apr 24, 2024

bors mentioned this pull request Apr 24, 2024
feat(ci): upload metrics to GitHub Artifacts ✨ #116517
Closed

Kobzol deleted the ci-dynamic-job branch April 24, 2024 09:07

This was referenced Apr 24, 2024
use precompiled rustc for non-dist builders #122709
Merged
Collapse -success and -failure jobs into an -outcome job #124014
Closed
Remove libc from MSVC targets #124050
Merged
DRAFT: Test building on Server 2022 #124174
Closed

Zalathar mentioned this pull request May 3, 2024
Don't build a broken/untested profiler runtime on mingw targets #122613
Merged

Kobzol mentioned this pull request May 6, 2024
Rewrite CI documentation rust-lang/rustc-dev-guide#1972
Merged

Kobzol mentioned this pull request May 24, 2024
Implement dynamic CI job selection on rust-lang/rustrust-lang/infra-team#104
Closed

Uh oh!

Uh oh!

Conversation

Kobzol commented Apr 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Kobzol commented Apr 19, 2024

Uh oh!

bors commented Apr 19, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

bors commented Apr 20, 2024

Uh oh!

emilyalbini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kobzol commented Apr 23, 2024

Uh oh!

emilyalbini commented Apr 23, 2024

Uh oh!

bors commented Apr 23, 2024

Uh oh!

Kobzol commented Apr 23, 2024

Uh oh!

RalfJungApr 23, 2024

Choose a reason for hiding this comment

Uh oh!

KobzolApr 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bors commented Apr 24, 2024

Uh oh!

bors commented Apr 24, 2024

Uh oh!

Kobzol commented Apr 24, 2024

Uh oh!

rust-timer commented Apr 24, 2024

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Kobzol commented Apr 19, 2024•
edited
Loading

KobzolApr 23, 2024•
edited
Loading