[wasm] Rationalize wasm jobs run on CI by radical · Pull Request #65035 · dotnet/runtime

radical · 2022-02-09T00:00:01Z

Essentially:

For rolling builds, we want to run all the wasm jobs, unconditionally
- And runtime-extra-platforms runs the tests that were not in runtime, and runtime-staging
For PRs, we use the various path checks

Changes:

split wasm jobs into various templates
combine, and simplify some jobs

Updated state of wasm jobs

PR:

runtime-extra-platforms, and runtime-wasm run only when manually triggered with a comment - /azp run <pipeline-name>
runtime, and runtime-staging, run jobs only when relevant paths change. And for EAT, and AOT, only smoke tests are run.
And when runtime-wasm is triggered manually, it runs all the wasm jobs completely

.	runtime	runtime-staging	runtime-extra-platforms(manual only)	runtime-wasm (manual only)
libtests	linux: all, only-pc	windows: all, only-pc	linux+windows: all, only-pc	linux+windows: all, always
libtests eat	linux: smoke, only-pc	-	linux: all, only-pc	linux: all, always
libtests aot	linux: smoke, only-pc	windows: smoke, only-pc	linux+windows: all, only-pc	linux+windows: all, always

Wasm.Build.Tests	linux: only-pc	windows: only-pc	linux+windows: only-pc	linux+windows
Debugger tests	-	linux+windows: only-pc	linux+windows: only-pc	linux+windows
Runtime tests	linux: only-pc	-	linux: only-pc	linux

Rolling build (twice a day):

runtime, and runtime-staging, run all the wasm jobs unconditionally, but EAT, and AOT still run only smoke tests.
runtime-extra-platforms also runs by default. And it runs only the cases not covered by the above two pipelines.
jobs w/o only-pc are always run

.	runtime	runtime-staging	runtime-extra-platforms (always run)	runtime-wasm (manual only)
libtests	linux: all(v8/chr)	windows: all	none	N/A
libtests eat	linux: smoke	-	linux: all
libtests aot	linux: smoke	windows: smoke	linux+windows: all

Wasm.Build.Tests	linux: always	windows: always	none
Debugger tests	-	linux+windows: always	none
Runtime tests	linux: always	-	none

ghost · 2022-02-09T00:00:08Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Essentially:

For rolling builds, we want to run all the wasm jobs, unconditionally
For PRs, we use the various path checks

Changes:

runtime-staging: for rolling builds, always run the wasm jobs
runtime-extra-platforms: for rolling builds, don't run any wasm jobs, as they are now run unconditionally by runtime, and runtime-staging pipelines
combine, and simplify some jobs

Author:	radical
Assignees:	radical
Labels:	`area-Infrastructure-mono`
Milestone:	-

radical · 2022-02-09T00:05:30Z

/azp run runtime-wasm

azure-pipelines · 2022-02-09T00:05:43Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-09T03:28:01Z

Two ways to do this:

For rolling builds, build all the wasm jobs in runtime+runtime-staging
- this is what I tried to implement here, but I didn't realize that runSmokeTestsOnly will be true for runtime+runtime-staging, even on rolling builds. So, we would need to make it true only on PRs.
For rolling builds, don't run any wasm jobs in runtime+runtime-staging, instead run all of them in runtime-extra-platforms

Which one makes sense?

radical · 2022-02-09T21:57:34Z

@safern any thoughts on #65035 (comment) ?

safern · 2022-02-10T00:45:38Z

@safern any thoughts on #65035 (comment) ?

This goes against the goal that we had with adding runtime-extra-platforms pipeline, the goal and intent is that runtime PR builds are the same as runtime rolling builds, so if we run smoke tests on PRs, the rolling build should run the same set of tests. This helps CI Council and infrastructure team evaluate and investigate pass rate for the pipeline and know in what state our PRs are in.

I would try and do the same for runtime-extra-platforms as rolling builds is our only input of data on how stable a pipeline is (this is because PR builds can fail due to a change on the PR itself, not because the pipeline is broken in that branch).

radical · 2022-02-15T02:04:11Z

/azp run runtime-wasm

azure-pipelines · 2022-02-15T02:04:27Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-15T05:31:10Z

Need to confirm what runtime-extra-platforms, and runtime-wasm would run on the PR.

radical · 2022-02-15T05:31:19Z

/azp run runtime-extra-platforms

radical · 2022-02-15T05:31:27Z

/azp run runtime-wasm

azure-pipelines · 2022-02-15T05:31:40Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2022-02-15T05:31:41Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-15T07:16:59Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T07:17:29Z

Azure Pipelines successfully started running 2 pipeline(s).

radical · 2022-02-15T07:19:50Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T07:20:25Z

Azure Pipelines successfully started running 2 pipeline(s).

radical · 2022-02-15T07:52:29Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T07:53:01Z

Azure Pipelines successfully started running 2 pipeline(s).

radical · 2022-02-15T08:16:30Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T08:17:02Z

Azure Pipelines successfully started running 2 pipeline(s).

melvinmezzy

Feedback

radical · 2022-02-15T08:33:39Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T08:34:13Z

Azure Pipelines successfully started running 2 pipeline(s).

radical · 2022-02-15T08:42:14Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T09:40:03Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-15T16:35:06Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-15T16:35:42Z

Azure Pipelines successfully started running 2 pipeline(s).

Essentially: - For rolling builds, we want to run all the wasm jobs, unconditionally - And `runtime-extra-platforms` runs the tests that were not in `runtime`, and `runtime-staging` - For PRs, we use the various path checks Changes: - split wasm jobs into various templates - combine, and simplify some jobs * `runtime-extra-platforms`, and `runtime-wasm` run only when manually triggered with a comment - `/azp run <pipeline-name>` * `runtime`, and `runtime-staging`, run jobs only when relevant paths change. And for `EAT`, and `AOT`, only smoke tests are run. * And when `runtime-wasm` is triggered manually, it runs *all* the wasm jobs completely | . | runtime | runtime-staging | runtime-extra-platforms(manual only) | runtime-wasm (manual only) | | ----------------- | -------------------- | --------------- | ------------------------------------ | ------- | | libtests | linux: all, only-pc | windows: all, only-pc | linux+windows: all, only-pc | linux+windows: all, always | | libtests eat | linux: smoke, only-pc | - | linux: all, only-pc | linux: all, always | | libtests aot | linux: smoke, only-pc | windows: smoke, only-pc | linux+windows: all, only-pc | linux+windows: all, always | | | | | | | | Wasm.Build.Tests | linux: only-pc | windows: only-pc | linux+windows: only-pc | linux+windows | | Debugger tests | - | linux+windows: only-pc | linux+windows: only-pc | linux+windows | | Runtime tests | linux: only-pc | - | linux: only-pc | linux | * `runtime`, and `runtime-staging`, run all the wasm jobs unconditionally, but `EAT`, and `AOT` still run only smoke tests. * `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by the above two pipelines. * jobs w/o `only-pc` are always run | . | runtime | runtime-staging | runtime-extra-platforms (always run) | runtime-wasm (manual only) | | ----------------- | ------------- | --------------- | ------------------------------------ | ------ | | libtests | linux: all(v8/chr/node) | windows: all | linux+windows: all with node | N/A | | libtests eat | linux: smoke | - | linux: all | | | libtests aot | linux: smoke | windows: smoke | linux+windows: all | | | | | | | | | Wasm.Build.Tests | linux: always | windows: always | none | | | Debugger tests | - | linux+windows: always | none | | | Runtime tests | linux: always | - | none | |

radical · 2022-02-15T20:34:33Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-02-15T20:34:55Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2022-02-16T22:50:34Z

/azp run runtime-wasm,runtime-extra-platforms

azure-pipelines · 2022-02-16T22:51:06Z

Azure Pipelines successfully started running 2 pipeline(s).

# Conflicts: # eng/pipelines/runtime.yml

radical · 2022-02-17T01:56:25Z

Merged main to resolve new conflicts.

lewing

Any obections?

radical · 2022-02-17T21:27:22Z

This is essentially moving the wasm jobs to separate template files, and then using them in the original locations.

radical · 2022-02-18T16:16:23Z

Wasm.Build.Tests haven't run in more than 14days now, this fixes the case.

ghost added the area-Infrastructure-mono label Feb 9, 2022

ghost assigned radical Feb 9, 2022

radical added the NO-REVIEW Experimental/testing PR, do NOT review it label Feb 9, 2022

radical force-pushed the wasm-ci branch from dd4faee to cafee75Compare February 12, 2022 03:08

radical force-pushed the wasm-ci branch from a674d40 to ac44549Compare February 15, 2022 05:22

melvinmezzy approved these changes Feb 15, 2022
View reviewed changes

Comment threadeng/pipelines/common/templates/wasm-library-tests.yml Outdated
Comment threadeng/pipelines/common/templates/wasm-library-tests.yml Outdated

runfoappBot mentioned this pull request Feb 15, 2022
System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter failing on Windows Server 2022 #65026
Closed

radical force-pushed the wasm-ci branch from 79175d0 to 6c8b2feCompare February 15, 2022 20:33

radical force-pushed the wasm-ci branch from 6c8b2fe to 35da22aCompare February 15, 2022 21:00

radical requested review from akoeplinger and safern February 15, 2022 21:00

radical marked this pull request as ready for review February 15, 2022 21:00

radical requested a review from marek-safar as a code owner February 15, 2022 21:00

radical removed the NO-REVIEW Experimental/testing PR, do NOT review it label Feb 15, 2022

radical requested a review from lewing February 15, 2022 21:06

radical added 2 commits February 16, 2022 20:51

Merge remote-tracking branch 'origin/main' into wasm-ci
cb46096
# Conflicts: # eng/pipelines/runtime.yml

[wasm] Run tests with node only on runtime-wasm
09a3574

lewing approved these changes Feb 17, 2022
View reviewed changes

runfoappBot mentioned this pull request Feb 17, 2022
System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync failing in runtime-extra-platforms #65021
Closed

radical merged commit 463bd75 into dotnet:mainFeb 18, 2022

radical deleted the wasm-ci branch February 18, 2022 16:16

ghost locked as resolved and limited conversation to collaborators Mar 20, 2022

Uh oh!

Conversation

radical commented Feb 9, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Updated state of wasm jobs

PR:

Rolling build (twice a day):

Uh oh!

ghost commented Feb 9, 2022

Uh oh!

radical commented Feb 9, 2022

Uh oh!

azure-pipelinesBot commented Feb 9, 2022

Uh oh!

radical commented Feb 9, 2022

Uh oh!

radical commented Feb 9, 2022

Uh oh!

safern commented Feb 10, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

melvinmezzy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 15, 2022

Uh oh!

azure-pipelinesBot commented Feb 15, 2022

Uh oh!

radical commented Feb 16, 2022

Uh oh!

azure-pipelinesBot commented Feb 16, 2022

Uh oh!

radical commented Feb 17, 2022

Uh oh!

lewing left a comment

Choose a reason for hiding this comment

Uh oh!

radical commented Feb 9, 2022•
edited
Loading