Use a single pipeline with runtime conditioned jobs in PRs by safern · Pull Request #1473 · dotnet/runtime

safern · 2020-01-08T23:28:08Z

This PR deletes all the individual per-partition builds and adds all the builds to runtime.yml and conditions tests and builds based on paths changed in a PR.

The result of this PR is as follows:

CoreCLR Builds

Build	PR	Rolling CI
Build CoreCLR OSX x64 Checked	Only CoreCLR or Libraries Changes	Always
Build CoreCLR Linux x64 Checked	Only CoreCLR Changes	Always
Build CoreCLR Linux arm Checked	Only CoreCLR Changes	Always
Build CoreCLR Linux arm64 Checked	Only CoreCLR Changes	Always
Build CoreCLR Linux musl x64 Checked	Only CoreCLR Changes	Always
Build CoreCLR Windows NT x64 Checked	Only CoreCLR Changes	Always
Build CoreCLR Windows NT x86 Checked	Only CoreCLR Changes	Always
Build CoreCLR Windows NT arm Checked	Only CoreCLR Changes	Always
Build CoreCLR Windows NT arm64 Checked	Only CoreCLR Changes	Always
Build CoreCLR OSX x64 Release	Always	Always
Build CoreCLR Linux x64 Release	Always	Always
Build CoreCLR Linux arm Release	Always	Always
Build CoreCLR Linux arm64 Release	Always	Always
Build CoreCLR Linux musl x64 Release	Always	Always
Build CoreCLR Linux musl arm64 Release	Always	Always
Build CoreCLR Windows NT x64 Release	Always	Always
Build CoreCLR Windows NT x86 Release	Always	Always
Build CoreCLR Windows NT arm Release	Always	Always
Build CoreCLR Windows NT arm64 Release	Always	Always
Formatting Linux x64	Only CoreCLR Changes	Always
Formatting Windows NT x64	Only CoreCLR Changes	Always

Libraries Builds

Build	PR	Rolling CI
Build Libraries OSX x64 Debug	Always	Never
Build Libraries OSX x64 Release	Never	Always
Build Libraries Linux x64 Debug	Always	Never
Build Libraries Linux x64 Release	Never	Always
Build Libraries Linux arm Release	Always	Always
Build Libraries Linux arm64 Debug	Always	Never
Build Libraries Linux arm64 Release	Never	Always
Build Libraries Linux musl x64 Debug	Always	Never
Build Libraries Linux musl x64 Release	Never	Always
Build Libraries Linux musl arm64 Release	Always	Always
Build Libraries Windows NT x64 Debug	Always	Never
Build Libraries Windows NT x64 Release	Never	Always
Build Libraries Windows NT x86 Release	Always	Always
Build Libraries Windows NT x86 Debug	Only Libraries Changes	Never
Build Libraries Windows NT arm Release	Always	Always
Build Libraries Windows NT arm64 Release	Always	Always
Build Libraries WebAssembly wasm Debug	Only Libraries Changes	Never
Build Libraries WebAssembly wasm Release	Never	Always
Build Libraries All Configurations x64 Debug	Only Libraries Changes	Never
Build Libraries All Configurations x64 Release	Never	Always
Build Libraries NETFX x86 Release	On Libraries Changes	Always
Build Libraries NETFX x64 Release	Never	Always

Installer Build and Test

Build	PR	Rolling CI
Installer Build and Test OSX x64 Debug	Always	Never
Installer Build and Test OSX x64 Release	Never	Always
Installer Build and Test Linux x64 Debug	Always	Never
Installer Build and Test Linux x64 Release	Never	Always
Installer Build and Test Linux arm Debug	Always	Never
Installer Build and Test Linux arm Release	Never	Always
Installer Build and Test Linux arm64 Debug	Always	Never
Installer Build and Test Linux arm64 Release	Never	Always
Installer Build and Test Linux musl x64 Debug	Always	Never
Installer Build and Test Linux musl x64 Release	Never	Always
Installer Build and Test Linux musl arm64 Debug	Always	Never
Installer Build and Test Linux musl arm64 Release	Never	Always
Installer Build and Test Windows NT x64 Debug	Always	Never
Installer Build and Test Windows NT x64 Release	Never	Always
Installer Build and Test Windows NT x86 Debug	Always	Never
Installer Build and Test Windows NT x86 Release	Never	Always
Installer Build and Test Windows NT arm Debug	Always	Never
Installer Build and Test Windows NT arm Release	Never	Always
Installer Build and Test Windows NT arm64 Debug	Always	Never
Installer Build and Test Windows NT arm64 Release	Never	Always

CoreCLR Test Builds

Build	PR	Rolling CI
CoreCLR CrossGen Linux arm Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build OSX x64 Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Linux x64 Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Linux arm Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Linux arm64 Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Windows NT x64 Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Windows NT x86 Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Windows NT arm Checked	Only CoreCLR Changes	Always
CoreCLR Pri0 Test Build Windows NT arm64 Checked	Only CoreCLR Changes	Always

Libraries Test Build

Build	PR	Rolling CI
Libraries Test Build OSX x64 Debug	CoreCLR or Libraries Changes	Never
Libraries Test Build OSX x64 Release	Never	Always
Libraries Test Build Linux x64 Debug	CoreCLR or Libraries Changes	Never
Libraries Test Build Linux x64 Release	Never	Always
Libraries Test Build Windows NT x64 Debug	CoreCLR or Libraries Changes	Never
Libraries Test Build Windows NT x64 Release	Never	Always

CoreCLR Test Runs

Build	PR	Rolling CI
CoreCLR Pri0 Test Run OSX x64 Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Linux x64 Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Linux arm Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Linux arm64 Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Windows NT x64 Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Windows NT x86 Checked	CoreCLR Changes	Always
CoreCLR Pri0 Test Run Windows NT arm Checked	CoreCLR Changes	Always
(Removed, not enough hardware) CoreCLR Pri0 Test Run Windows NT arm64 Checked	CoreCLR Changes	Always

Libraries Test Runs (Release CoreCLR)

Build	PR	Rolling CI
Libraries Test Run OSX x64 Debug	Libraries Changes	Never
Libraries Test Run OSX x64 Release	Never	Always
Libraries Test Run Linux x64 Debug	Libraries Changes	Never
Libraries Test Run Linux x64 Release	Never	Always
Libraries Test Run Linux arm Release	Never	Always
Libraries Test Run Linux arm64 Release	Never	Always
Libraries Test Run Linux musl x64 Debug	Libraries Changes	Never
Libraries Test Run Linux musl x64 Release	Never	Always
Libraries Test Run Linux musl arm64 Release	Never	Always
Libraries Test Run Windows NT x64 Debug	Libraries Changes	Never
Libraries Test Run Windows NT x64 Release	Never	Always
Libraries Test Run Windows NT x86 Release	Libraries Changes	Always
Libraries Test Run Windows NT x86 Debug	Libraries Changes	Never

Libraries Test Run (Checked CoreCLR)

Build	PR	Rolling CI
Libraries Test Run OSX x64 Debug	CoreCLR Changes or Libraries Changes	Never
Libraries Test Run OSX x64 Release	Never	Always
Libraries Test Run Linux x64 Debug	CoreCLR Changes	Never
Libraries Test Run Linux x64 Release	Never	Always
Libraries Test Run Linux musl x64 Debug	CoreCLR Changes	Never
Libraries Test Run Linux musl x64 Release	Never	Always
Libraries Test Run Windows NT x64 Debug	CoreCLR Changes	Never
Libraries Test Run Windows NT x64 Release	Never	Always
Libraries Test Run Windows NT x86 Release	CoreCLR Changes	Always
(Disabled) Libraries Test Run Windows NT arm Release	CoreCLR Changes	Always
Libraries Test Run Windows NT arm64 Release	Never	Always

cc: @jkotas @stephentoub @danmosemsft @BruceForstall

safern · 2020-01-15T19:05:14Z

Doesn't that mean that when only touching libraries you increase the CI time significantly?

Actually from my tests I've seen that running tests against a Checked runtime takes similar time to release, so if we run tests for OSX x64 and Windows x86 it shouldn't increase time significantly. Specially because we build libraries tests once for each OS, so that saves some time. I think it is worth the try, we can always remove them, but is good extra coverage.

ViktorHofer · 2020-01-15T19:08:59Z

Actually from my tests I've seen that running tests against a Checked runtime takes similar time to release

Looks like you are right. In coreclr release/3.1 on a random build, corefx checked takes about 1h 10min: https://dev.azure.com/dnceng/public/_build/results?buildId=483366&view=logs&jobId=dbf66e29-15d1-5c85-f25f-a8bda8f9b70c.

dagood

Infra LGTM. Just a formatting suggestion--my usual hatred of scrolling horizontally. 😄

safern · 2020-01-15T19:14:16Z

Looks like you are right. In coreclr release/3.1 on a random build, corefx checked takes about 1h 10min

Right and I believe that's dependent on how long it takes to provision a helix machines, for example in this run for this current PR it took 11 mins.
https://dev.azure.com/dnceng/public/_build/results?buildId=481127

BruceForstall · 2020-01-15T19:37:34Z

Looks like you're matrix is missing:

CoreCLR Pri0 Test Build Linux arm64 Checked
CoreCLR Pri0 Test Run Linux arm64 Checked

There are a lot of CoreCLR Release builds. Are these required for some non-CoreCLR test runs? They aren't used for any CoreCLR test runs, and it seems like we could save CI resources by only doing one or two as Release build spot-checks in PRs unless they are otherwise required.

There is no

Build Libraries Linux arm Debug
Build Libraries Linux musl arm64 Debug
Build Libraries Windows NT x86 Debug
Build Libraries Windows NT arm Debug
Build Libraries Windows NT arm64 Debug

in the table. Is that an intentional pruning of the full matrix?

safern · 2020-01-15T19:46:47Z

There are a lot of CoreCLR Release builds. Are these required for some non-CoreCLR test runs?

Yes these are required to run libraries and installer tests... we want to run tests against a release CoreCLR. We will only use the Checked ones for CoreCLR changes, however as discussed here, I will add a couple that will also happen when there are libraries changes.

However, I will see in the test matrix for installer and libraries and see if we can prune some CoreCLR Release builds.

Looks like you're matrix is missing:

Thanks 😄

Is that an intentional pruning of the full matrix?

Yes, we try to balance out the Debug/Release builds we do by covering in other arches, OSs, I basically just ported over what we've been testing in Libraries from the corefx world.

ViktorHofer · 2020-01-15T20:43:52Z

However, I will see in the test matrix for installer and libraries and see if we can prune some CoreCLR Release builds.

Just my 5c. I don't think we should trim builds further. I would even go into the opposite direction and always build everything (still with a good mix of Debug vs Release). With the single pipeline we are already saving costs in comparison to what we had before if you combine coreclr + corefx + core-setup. Running only a specific set of tests (libraries vs coreclr) makes sense as that's the most resource intensive phase.

jaredpar · 2020-01-15T21:08:22Z

. I would even go into the opposite direction and always build everything (still with a good mix of Debug vs Release).

We need to be careful with builds that use hardware resources like ARM. These are much more limited than our VM builds. The intent going into this is we would exclude ARM unless CLR changed to ensure they had sufficient resources to make progress on ARM specific changes.

ViktorHofer · 2020-01-15T21:31:47Z

The intent going into this is we would exclude ARM unless CLR changed to ensure they had sufficient resources to make progress on ARM specific changes.

AFAIK we always want to build on arm but limit testing on it.

safern · 2020-01-15T21:34:23Z

AFAIK we always want to build on arm but limit testing on it.

Building on arm doesn't take any arm resources, so it's fine to build for arm, for windows we build on x64 machines but target arm as the architecture. For Linux, we use docker images with a cross rootfs directory that contains all the arm binaries, and build using that as the root file system.

BruceForstall · 2020-01-15T21:38:08Z

I'm fine with building everything as long as it doesn't cost too many resources we care about, doesn't increase long pole times, and doesn't cause resource exhaustion when many PR's happen simultaneously. With one caveat: the more jobs we have, the more likely it seems we will have a "random" failure in one or more of the jobs. With so many moving parts in this system, this seems inevitable. This because super frustrating for users.

safern · 2020-01-15T21:41:34Z

Yeah I totally agree and I tried to keep the matrix as minimal as possible, but building release coreclr is required to run libraries and installer tests. I think I came up with the minimal set of the matrix.

We could evaluate, but something that we could do is add more conditions and for example if coreclr is the only partition touched, only run libraries and installer tests on a checked runtime and that would avoid building release coreclr, but I thought that was more prone to errors and missing the right coverage that we've been having for years already.

safern · 2020-01-16T01:12:03Z

CoreCLR failure is: https://github.com/dotnet/coreclr/issues/26241
Libraries failure is: #131

I added a commit to disable the Libraries test on a checked runtime: db5e58b

Merging as everything else was green.

sandreenko · 2020-01-16T01:29:18Z

Thanks @safern , that looks great.

The only question is about "CoreCLR Pri0 Test Run Windows NT arm64 Checked job", I believe we do not run tests on arm64 windows (do not have enough machines). Historically that job existed only to build tests (because there was no separation between "Build tests" and "Run tests"), but now that probably should be deleted to avoid confusion.

josalem · 2020-01-16T01:34:23Z

CoreCLR failure should be resolved by #1794!

trylek · 2020-01-16T01:36:28Z

I believe we'll need a bit to iterate on the ideal platform combo. @jkotas is specifically asking for runs against checked builds to catch assert failures and I believe he's got a very valid point. I continue working with dnceng and DDFUN to make them provide us with reasonable HW.

safern · 2020-01-16T19:00:21Z

The only question is about "CoreCLR Pri0 Test Run Windows NT arm64 Checked job", I believe we do not run tests on arm64 windows

Right, it currently doesn't run any tests as it is disabled, same for arm (which @trylek is working on). I'll put up a PR to remove it from the test runs to not confuse people as it will basically just be a no-op.
https://github.com/dotnet/runtime/blob/master/eng/pipelines/coreclr/templates/helix-queues-setup.yml#L129

The GVM analysis within the compiler assumes we can do analysis on top of canonical forms and we get analysis holes if the canonical form doesn't exist at compile time. GVM dispatch at runtime might end up building new types as well, so the extra template isn't strictly just bloat but I do admit there's some laziness in just throwing the extra template in to fix the problem. This wasn't hittable outside reflection-free mode because we generate canonical forms of everything outside reflection-free mode anyway. As I was looking for a place to put this in, I realized the `TypeNeedsGVMTableEntries` logic needs to run on canonical types as well (those are not `ConstructedEETypeNode`), so I moved it to a common spot. That part was not necessary to fix the reported problem, but it provides a convenient spot to put the canonical type dependency. Fixesdotnet#1473.

safern force-pushed the SinglePipeline branch 13 times, most recently from de62bfc to bc6fe28Compare January 9, 2020 05:27

jaredpar reviewed Jan 9, 2020
View reviewed changes

Comment threadeng/pipelines/runtime.yml Outdated

jaredpar reviewed Jan 9, 2020
View reviewed changes

Comment threadeng/pipelines/runtime.yml Outdated

safern mentioned this pull request Jan 9, 2020
Relax some asserts in SpanHelpers.T.cs #1472
Closed

safern force-pushed the SinglePipeline branch 14 times, most recently from c20d9c3 to 105ce20Compare January 12, 2020 00:06

dagood approved these changes Jan 15, 2020
View reviewed changes

Comment threadeng/pipelines/runtime.yml Outdated

jaredpar reviewed Jan 15, 2020
View reviewed changes

Comment threadeng/pipelines/runtime.yml

PR Feedback
295de41

safern force-pushed the SinglePipeline branch from f4e0047 to 295de41Compare January 15, 2020 22:38

ViktorHofer reviewed Jan 15, 2020
View reviewed changes

Comment threadeng/pipelines/runtime.yml

safern mentioned this pull request Jan 15, 2020
Use the "live" NETCoreApp shared framework in CoreCLR and Libraries tests #487
Closed

Disable long running tests checked runtime
db5e58b

safern merged commit 2fdc85d into masterJan 16, 2020

safern deleted the SinglePipeline branch January 16, 2020 01:12

This was referenced Jan 16, 2020
Add root build definition which builds the entire repo #98
Closed
crossgen-comparison Linux arm checked fails with timeouts #1282
Closed

safern mentioned this pull request Jan 16, 2020
Add back asserts to catch unintentional use of generic SpanHelpers overloads #1767
Closed

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Uh oh!

Conversation

safern commented Jan 8, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CoreCLR Builds

Libraries Builds

Installer Build and Test

CoreCLR Test Builds

Libraries Test Build

CoreCLR Test Runs

Libraries Test Runs (Release CoreCLR)

Libraries Test Run (Checked CoreCLR)

Uh oh!

Uh oh!

Uh oh!

safern commented Jan 15, 2020

Uh oh!

ViktorHofer commented Jan 15, 2020

Uh oh!

dagood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

safern commented Jan 15, 2020

Uh oh!

Uh oh!

BruceForstall commented Jan 15, 2020

Uh oh!

safern commented Jan 15, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViktorHofer commented Jan 15, 2020

Uh oh!

jaredpar commented Jan 15, 2020

Uh oh!

ViktorHofer commented Jan 15, 2020

Uh oh!

safern commented Jan 15, 2020

Uh oh!

BruceForstall commented Jan 15, 2020

Uh oh!

safern commented Jan 15, 2020

Uh oh!

Uh oh!

safern commented Jan 16, 2020

Uh oh!

sandreenko commented Jan 16, 2020

Uh oh!

josalem commented Jan 16, 2020

Uh oh!

trylek commented Jan 16, 2020

Uh oh!

safern commented Jan 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

safern commented Jan 8, 2020•
edited
Loading

safern commented Jan 15, 2020•
edited
Loading