[wasm] Compile .bc->.o in parallel, before passing to the linker by radical · Pull Request #54053 · dotnet/runtime

radical · 2021-06-11T09:12:18Z

Currently, we use mono-aot-cross to compile the managed assemblies
into bitcode files. Then we pass the .bc files to emcc for linking.
We can save some amount of work, if precompile .bc files to .o
files, and pass them for linking.

This would also allow us to rebuild the .o files only when needed,
thus improving incremental builds.
Also, emscripten has recently removed some amount of parallelization, so
this allows us to do some pre-linking work in parallel, and speed up the
builds a little bit.

This is implemented via a new EmccCompile task, that is meant only
for compiling, and not as a general purpose emcc invocation task.

Also, this bumps msbuild used for tasks compilation to 16.9.0.

- this cleans up `Utils.RunProcess` a bit, but doesn't change the behavior for the existing non-wasm users (Apple, and Android build tasks) - New overloads are added to support getting the output, and exit code, and throwing only when really needed - And running shell commands, appropriately on windows, and unix.

Currently, we use `mono-aot-cross` to compile the managed assemblies into bitcode files. Then we pass the `.bc` files to `emcc` for linking. We can save some amount of work, if precompile `.bc` files to `.o` files, and pass them for linking. This would also allow us to rebuild the `.o` files only when needed, thus improving incremental builds. Also, emscripten has recently removed some amount of parallelization, so this allows us to do some pre-linking work in parallel, and speed up the builds a little bit. This is implemented via a new `EmccCompile` task, that is meant *only* for compiling, and not as a general purpose `emcc` invocation task.

`System.IO.MemoryMappedFiles.Tests` - Issue: dotnet#54194

ghost · 2021-06-15T00:23:50Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	radical
Assignees:	-
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

- `IBuildEngine9.RequestCores` - This is available only in 16.10+, so bump msbuild used for building the tests

radical · 2021-06-16T17:07:48Z

@akoeplinger Does the msbuild bump to 16.10.0 look fine?

radical · 2021-06-16T21:38:04Z

/azp run runtime,runtime-staging

azure-pipelines · 2021-06-16T21:38:38Z

Azure Pipelines successfully started running 2 pipeline(s).

…third

lewing · 2021-06-17T00:39:44Z

/azp run runtime

- This is meant to be disabled for CI (it has `IgnoreForCI=true` in the csproj)

.. because we keep running into the following on browser, and other platforms: ``` /__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj : error NU1201: Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0) [/__w/1/s/Build.proj] \#\#[error]src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj(0,0): error NU1201: (NETCORE_ENGINEERING_TELEMETRY=Restore) Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0) ```

This is already not being sent to CI with `IgnoreForCI=true`. It fails on CI with: ``` /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj] \##[error]/_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: (NETCORE_ENGINEERING_TELEMETRY=Build) Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' Mono.Linker.LinkerFatalErrorException: /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' ---> System.NotImplementedException: switch at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.IsConstantBranch(OpCode opCode, Int32 operand) in illink.dll:token 0x6000770+0x41 at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RemoveConditions() in illink.dll:token 0x6000768+0x22a at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RewriteBody() in illink.dll:token 0x6000766+0xa at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessStack() in illink.dll:token 0x60004e9+0x102 at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessMethod(MethodDefinition method) in illink.dll:token 0x60004e5+0x44 at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, Scope& scope) in illink.dll:token 0x600040e+0x9b at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0x24 --- End of inner exception stack trace --- at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0xaa at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue() in illink.dll:token 0x6000396+0xa at Mono.Linker.Steps.MarkStep.Process() in illink.dll:token 0x6000393+0x0 at Mono.Linker.Steps.MarkStep.Process(LinkContext context) in illink.dll:token 0x600038a+0x41 at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step) in illink.dll:token 0x6000264+0x0 at Mono.Linker.Pipeline.Process(LinkContext context) in illink.dll:token 0x6000263+0xf at Mono.Linker.Driver.Run(ILogger customLogger) in illink.dll:token 0x60000fd+0x20 Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink /__w/1/s/.dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj] \##[error].dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: (NETCORE_ENGINEERING_TELEMETRY=Build) Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. ```

radical · 2021-06-22T21:07:34Z

cc @ericstj @ViktorHofer for the change disabling Microsoft.NETCore.Platforms.Tests for wasm.

Issue: dotnet#54194

radical · 2021-06-23T05:28:39Z

1 test failure for wasm:

[04:55:37] info: browser: Console websocket connected.
[04:55:37] info: Arguments: --run,WasmTestRunner.dll,System.IO.Compression.Tests.dll,-notrait,category=IgnoreForCI,-notrait,category=OuterLoop,-notrait,category=failing
[04:55:37] fail: [out of order message from the browser]: http://127.0.0.1:34229/runtime.js 234:26 Uncaught TypeError: Cannot set property 'loaded_cb' of null
[05:10:37] fail: Application has finished with exit code TIMED_OUT but 0 was expected

@Daniel-Genkin is already looking at fixing this.
I think we can merge this.

lewing · 2021-06-23T05:44:36Z

@Daniel-Genkin is already looking at fixing this.
I think we can merge this.

fix should be in #54587

ViktorHofer · 2021-06-23T06:25:46Z

@ViktorHofer Any idea why a msbuild bump to 16.10.0 in this PR might be failing wasm/android/iOS/etc builds? That's the only non-wasm specific change here.

That very much sounds like NuGet/Home#10368. Usually you can fix this by adding the transitive dependencies, in this case in the Platforms.Tests.csproj file.

radical · 2021-06-23T16:53:46Z

The failures are all unrelated to this PR.

ViktorHofer · 2021-06-23T17:47:16Z

The failures are all unrelated to this PR.

Usually we link to issues that track the failures according to https://github.com/dotnet/runtime/blob/main/docs/issues-pr-management.md#common-policies. Can you please do so for the issues that you were seeing in the last build?

radical · 2021-06-23T18:09:03Z

Failures:

Libraries Test Run checked coreclr windows x86 Release - #54628
Libraries Test Run release mono OSX x64 Debug - #54624
Libraries Test Run release coreclr windows x64 Debug - #54627
Libraries Test Run release coreclr Linux x64 Debug - #54623

And for CoreCLR Pri0 Runtime Tests Run windows arm64 checked - profiler.gc, and JIT.Directed failed but I'm not able to access the logs for them.

…nitial_config * origin/main: (362 commits) [wasm][debugger] Reuse debugger-agent on wasm debugger (dotnet#52300) Put Crossgen2 in sync with dotnet#54235 (dotnet#54438) Move System.Object serialization to ObjectConverter (dotnet#54436) Move setting fHasVirtualStaticMethods out of sanity check section (dotnet#54574) [wasm] Compile .bc->.o in parallel, before passing to the linker (dotnet#54053) Change PathInternal.IsCaseSensitive to a constant (dotnet#54340) Make mono_polling_required a public symbol (dotnet#54592) Respect EventSource::IsSupported setting in more codepaths (dotnet#51977) Root ComActivator for hosting (dotnet#54524) Add ILLink annotations to S.D.Common related to DbConnectionStringBuilder (dotnet#54280) Fix finalizer issue with regions (dotnet#54550) Add support for multi-arch install locations (dotnet#53763) Update library testing docs page to reduce confusion (dotnet#54324) [FileStream] handle UNC and device paths (dotnet#54483) Update NetAnalyzers version (dotnet#54511) Added runtime dependency to fix the intermittent test failures (dotnet#54587) Disable failing System.Reflection.Tests.ModuleTests.GetMethods (dotnet#54564) [wasm] Move AOT builds from `runtime-staging` to `runtime` (dotnet#54577) Keep obj node for ArrayIndex. (dotnet#54584) Disable another failing MemoryCache test (dotnet#54578) ...

ghost added the area-Build-mono label Jun 11, 2021

radical added 2 commits June 14, 2021 20:09

radical mentioned this pull request Jun 15, 2021
[wasm][aot] Runtime crash in System.IO.MemoryMappedFiles.Tests#54194
Closed

Disable failing tests
1909fcc
`System.IO.MemoryMappedFiles.Tests` - Issue: dotnet#54194

radical force-pushed the wasm-bc-to-o-parallel branch from 376b844 to 1909fccCompare June 15, 2021 00:19

radical marked this pull request as ready for review June 15, 2021 00:20

radical requested a review from marek-safar as a code owner June 15, 2021 00:20

radical requested review from akoeplinger, lewing, mdh1418, steveisok and vargaz June 15, 2021 00:23

radical added the arch-wasm WebAssembly architecture label Jun 15, 2021

radical added 5 commits June 15, 2021 14:57

Merge remote-tracking branch 'origin/main' into wasm-bc-third
24338c6

[wasm] EmccCompile task: Use msbuild's new resource management API
e7b638c
- `IBuildEngine9.RequestCores` - This is available only in 16.10+, so bump msbuild used for building the tests

cleanup
9f23826

Correctly handle non-aot native case
f87f705

Merge remote-tracking branch 'origin/main' into wasm-bc-third
093e133

radical requested a review from rainersigwald June 16, 2021 17:08

radical added 2 commits June 16, 2021 16:43

Merge remote-tracking branch 'origin/main' into wasm-bc-third
d7f867e

Merge branch 'main' into wasm-bc-to-o-parallel
f562d2e

lewing approved these changes Jun 16, 2021
View reviewed changes

radical added 2 commits June 16, 2021 19:56

Merge remote-tracking branch 'origin/main' into wasm-bc-third
f20240b

Merge remote-tracking branch 'rf/wasm-bc-to-o-parallel' into wasm-bc-…
6e90a22
…third

radical added 4 commits June 21, 2021 15:55

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel
d816231

Disable Microsoft.NETCore.Platforms.Tests on wasm, for local builds too
1be69d6
- This is meant to be disabled for CI (it has `IgnoreForCI=true` in the csproj)

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel
76d370a

radical removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jun 22, 2021

radical added 3 commits June 22, 2021 12:36

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel
9374c61

undo test exclusion change related to the 16.10.0 bump
b7f52e4

radical added 2 commits June 22, 2021 17:30

[wasm] Disable System.IO.MemoryMappedFiles.Tests
38e4fb8
Issue: dotnet#54194

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel
ad2255c

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel
e677e8a

radical merged commit 0416c34 into dotnet:mainJun 23, 2021

radical deleted the wasm-bc-to-o-parallel branch June 23, 2021 16:54

runfoappBot mentioned this pull request Jun 24, 2021
[wasm] System.Net.Http functional test failing:GetAsync_ManyDifferentResponseHeaders_ParsedCorrectly#54655
Closed

karelz mentioned this pull request Jun 24, 2021
Connect_DualMode_DnsConnect_RetrievedEndPoints_Success - Assert ::1 #54677
Closed

radical mentioned this pull request Jun 28, 2021
[wasm] Compile bitcode files, and use the .o files for the linking step #50285
Closed

karelz mentioned this pull request Jun 30, 2021
System.Net tests failing with System.TimeoutException #54778
Closed

ghost locked as resolved and limited conversation to collaborators Jul 23, 2021

Navigation Menu

Uh oh!

[wasm] Compile .bc->.o in parallel, before passing to the linker - #54053

[wasm] Compile .bc->.o in parallel, before passing to the linker#54053
radical merged 22 commits into
dotnet:mainfrom
radical:wasm-bc-to-o-parallel

radical commented Jun 11, 2021•
edited
Loading

Uh oh!

ghost commented Jun 15, 2021

Uh oh!

radical commented Jun 16, 2021

Uh oh!

radical commented Jun 16, 2021

Uh oh!

azure-pipelinesBot commented Jun 16, 2021

Uh oh!

lewing commented Jun 17, 2021

Uh oh!

radical commented Jun 22, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

lewing commented Jun 23, 2021

Uh oh!

ViktorHofer commented Jun 23, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

ViktorHofer commented Jun 23, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

radical commented Jun 11, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 15, 2021

Uh oh!

radical commented Jun 16, 2021

Uh oh!

radical commented Jun 16, 2021

Uh oh!

azure-pipelinesBot commented Jun 16, 2021

Uh oh!

lewing commented Jun 17, 2021

Uh oh!

radical commented Jun 22, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

lewing commented Jun 23, 2021

Uh oh!

ViktorHofer commented Jun 23, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

ViktorHofer commented Jun 23, 2021

Uh oh!

radical commented Jun 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

radical commented Jun 11, 2021•
edited
Loading