[wasm] Wasm.Build.Tests: add support for shared builds by radical · Pull Request #49398 · dotnet/runtime

radical · 2021-03-09T23:50:44Z

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:
public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Note: In interest of lowering test run times, this builds the tests in Release config only, on CI.
But when running the tests locally, both Debug, and Release configs are used.

ghost · 2021-03-09T23:50:52Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:
public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Author:	radical
Assignees:	-
Labels:	`area-Infrastructure-mono`
Milestone:	-

ghost · 2021-03-09T23:51:24Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:
public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Author:	radical
Assignees:	-
Labels:	`arch-wasm`, `area-Infrastructure-mono`
Milestone:	-

radical · 2021-03-11T02:40:56Z

This will be rebased once #49446 is merged.

- Essentially, we want to share builds wherever possible. Example cases: - Same build, but run with different hosts like v8/chrome/safari, as separate test runs - Same build but run with different command line arguments - Sharing builds especially helps when we are AOT'ing, which is slow! - This is done by caching the builds with the key: `public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);` - Also, ` SharedBuildClassFixture` is added, so that the builds can be cleaned up after all the tests in a particular class have finished running. - Each test run gets a randomly generated test id. This is used for creating: 1. build paths, like `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict` 2. and the log for running with xharness, eg. for Chrome, are in `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/` - split `WasmBuildAppTest.cs` into : `BuildTestBase.cs`, and `MainWithArgsTests.cs`.

…zation

.. tests. Code stolen from @maximlipin's dotnet#49204

For AOT we generate `pinvoke-table.h` in the obj directory. But there is one present in the runtime pack too. In my earlier changes the order in which these were passed as include search paths was changed from: `"-I/runtime/pack/microsoft.netcore.app.runtime.browser-wasm/Release/runtimes/browser-wasm/native/include/wasm" "-Iartifacts/obj/mono/Wasm.Console.Sample/wasm/Release/browser-wasm/wasm/"` .. which meant that the one from the runtime pack took precedence, and got used. So, fix the order! And change the property names to indicate where they are sourced from.

The environment variable is set on helix. During local testing it can be useful when using a locally built xharness.

radical · 2021-03-12T17:31:50Z

Unrelated test failures.

steveisok

LGTM. I'm curious the average helix run time w/ these changes.

radical · 2021-03-16T16:32:01Z

AFAICS, the test time stays at about 45mins. With this one we actually are building only Release, but #49559 will enable testing for Debug too. And I will add support to enable more parallelization.

Though when I look at the run times for the individual runs for wasm.build tests, they seem to be around ~25mins.

radical · 2021-03-16T17:09:58Z

AFAICS, the test time stays at about 45mins.

The helix step runs in total 45mins when I looked at one of the earlier main builds. The wasm.build test run itself takes ~25mins, it runs in parallel to the library tests.

radical · 2021-03-16T19:52:54Z

Repeatedly getting test run failures:

.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.21155.1/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) RestApiException`1: The response contained an invalid status code 500 Internal Server Error
Body: {"Message":"An error occured.","ActivityId":"0ade13a01812c1468a790d153e2fc3a0"}
at Microsoft.DotNet.Helix.Client.Storage.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Storage.cs:line 202
at Microsoft.DotNet.Helix.Client.Storage.NewAsync(ContainerCreationRequest body, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Storage.cs:line 164
at Microsoft.DotNet.Helix.Client.ApiBlobHelper.GetContainerAsync(String requestedName, String targetQueue, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/StorageHelpers/ApiBlobHelper.cs:line 22
at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 157
at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.Dot

.. tracked in https://github.com/dotnet/core-eng/issues/12499 .

radical · 2021-03-17T16:33:59Z

Added a few more tests, and some cleanup.

radical · 2021-03-26T02:22:35Z

investigating the test run time

TLDR; - this might help with the job getting scheduled first, and thus having a chance at completing at the same time as others. Reasoning: The problem this is trying to fix is: 1. The helix step submits 3 jobs: a. library tests to be run with v8 b. library tests to be run with browser (scenario=wasmtestonbrowser) c. Wasm.Build.Tests (scenario=buildwasmapps) 2. The 3 jobs, individually take roughly 30mins each 3. And they get submitted at roughly the same time 4. But .. the first two seem to complete earlier, and the 3rd one completes 25-30mins later. The hypothesis is that all the machines might be busy processing the 200+ work items from each of the first two steps, and so Wasm.Build.Tests get scheduled pretty late. So, here we move that to be submitted first, in the hope that it would be able to run in parallel with the other jobs.

radical · 2021-03-26T19:48:56Z

@akoeplinger with your suggested change (7caed5a) it ran in 43mins, instead of 59! 👍

radical · 2021-03-26T22:21:48Z

@lewing @steveisok this is ready for another look! And hopefully merge :D

ghost added the area-Infrastructure-mono label Mar 9, 2021

radical marked this pull request as draft March 9, 2021 23:51

radical added the arch-wasm WebAssembly architecture label Mar 9, 2021

radical force-pushed the shared-wasm-build-tests branch 3 times, most recently from 3f95a20 to 78840c3Compare March 10, 2021 21:52

radical marked this pull request as ready for review March 10, 2021 22:43

radical requested a review from marek-safar as a code owner March 10, 2021 22:43

radical requested review from akoeplinger, lewing and steveisok March 10, 2021 22:44

radical force-pushed the shared-wasm-build-tests branch from 5a03409 to cedeecaCompare March 11, 2021 02:35

radical force-pushed the shared-wasm-build-tests branch from cedeeca to 2dbfc6aCompare March 11, 2021 05:36

radical added 9 commits March 11, 2021 17:35

[wasm] Wasm.Build.Tests: Add test for relinking with InvariantGlobali…
af44fea
…zation

[wasm] Wasm.Build.Tests - Check CultureInfo for invariant culture
1b102b3
.. tests. Code stolen from @maximlipin's dotnet#49204

fix invariant+aot test
33d87f2

[wasm] Only test with Release config on CI
250653e

[wasm] Fallback to dotnet xharness if XHARNESS_CLI_PATH is not set.
3cd29e0
The environment variable is set on helix. During local testing it can be useful when using a locally built xharness.

[wasm] fix invariant test - 'en-ES' -> 'es-ES'
739881f

[wasm] RunWithEmSdkEnv: log the working directory also
fcf0da6

radical force-pushed the shared-wasm-build-tests branch from 808e761 to fcf0da6Compare March 11, 2021 23:52

radical added 2 commits March 11, 2021 19:18

Merge remote-tracking branch 'origin/main' into wasm-build-tests
cb7aff7

[wasm] Re-enable wasm build tests
1ce676f

runfoappBot mentioned this pull request Mar 12, 2021
Test failure: System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests/CreateChain_Hybrid #25979
Closed

radical added 2 commits March 16, 2021 10:45

[wasm] Add regression test for issuedotnet#49588
522424a

Merge remote-tracking branch 'origin/main' into wasm-build-tests
86b4b36

steveisok approved these changes Mar 16, 2021
View reviewed changes

fix test
aae95e2

radical added 2 commits March 17, 2021 12:30

[wasm] Cleanup, and add more tests
3540462

Merge remote-tracking branch 'origin/main' into wasm-build-tests
5e90d6f

radical added 4 commits March 17, 2021 13:16

Merge remote-tracking branch 'origin/main' into wasm-build-tests
02045a3

Update tests to track wasm relinking being default in some cases
c21da91

Fix InvariantGlobalization to track change in wasm relinking defaults
d6e9732

Merge remote-tracking branch 'origin/main' into wasm-build-tests
da64faf

runfoappBot mentioned this pull request Mar 22, 2021
System.Security.Cryptography.X509Certificates.Tests.ChainTests.BuildInvalidSignatureTwice #48851
Closed

radical added 3 commits March 25, 2021 10:46

Merge remote-tracking branch 'origin/main' into wasm-build-tests
eaf1615

[wasm] Update emsdk check message to track changes
97b83b8

[wasm] Update tests to track changes
f463714

steveisok approved these changes Mar 26, 2021
View reviewed changes

radical mentioned this pull request Mar 26, 2021
[wasm] Fix order of include paths, to have the obj dir first #50303
Merged

radical merged commit 617a18d into dotnet:mainMar 29, 2021

radical deleted the shared-wasm-build-tests branch March 29, 2021 19:33

ghost locked as resolved and limited conversation to collaborators Apr 28, 2021

karelz added this to the 6.0.0 milestone May 20, 2021

Navigation Menu

Uh oh!

[wasm] Wasm.Build.Tests: add support for shared builds - #49398

[wasm] Wasm.Build.Tests: add support for shared builds#49398
radical merged 24 commits into
dotnet:mainfrom
radical:shared-wasm-build-tests

radical commented Mar 9, 2021•
edited
Loading

Uh oh!

ghost commented Mar 9, 2021

Uh oh!

ghost commented Mar 9, 2021

Uh oh!

radical commented Mar 11, 2021

Uh oh!

radical commented Mar 12, 2021

Uh oh!

steveisok left a comment

Uh oh!

radical commented Mar 16, 2021

Uh oh!

radical commented Mar 16, 2021•
edited
Loading

Uh oh!

radical commented Mar 16, 2021

Uh oh!

radical commented Mar 17, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

radical commented Mar 9, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Mar 9, 2021

Uh oh!

ghost commented Mar 9, 2021

Uh oh!

radical commented Mar 11, 2021

Uh oh!

radical commented Mar 12, 2021

Uh oh!

steveisok left a comment

Choose a reason for hiding this comment

Uh oh!

radical commented Mar 16, 2021

Uh oh!

radical commented Mar 16, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radical commented Mar 16, 2021

Uh oh!

radical commented Mar 17, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

radical commented Mar 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radical commented Mar 9, 2021•
edited
Loading

radical commented Mar 16, 2021•
edited
Loading