[WASM] Adding a new switch to run wasm on NodeJS

[Daniel-Genkin]

This PR adds the missing pieces for NodeJS support to MONO WASM!

How does it work

The main differences between this and running the console sample regularly in v8 is that dotnet.js is modularized so we use require to load it. We now also have a package.json and package-lock.json files as well as the node_modules folder. Besides that, most changes to the code were related to various NodeJS related nuances such as the scoping of this but are ignored by the browser.

These changes are triggered by specifying /p:ForNode=true when building and publishing. If this switch is not specified then it is false.

What has changed

• New /p:ForNode=true switch to toggle the extra steps needed for the NodeJS build
• 4 New CI Pipelines for maintaining and testing in NodeJS
• In the XHarness repo I made a PR Added NodeJS Engine xharness#665 to create a new NodeJS engine, so now when running tests we can use the /p:JSEngine=NodeJS switch

How to run locally

1. ./build.cmd mono+libs -os Browser -c Debug /p:ForNode=true (-c Release also works)
2. ./dotnet.cmd publish /p:TargetArchitecture=wasm /p:TargetOS=Browser /p:ForNode=true .\src\mono\sample\wasm\console\Wasm.Console.Sample.csproj -c Debug
3. cd src\mono\sample\wasm\console\bin\Debug\AppBundle
4. Run via npm test, ./run-node.sh (or run-node.cmd if on windows) or node runtime.js --run Wasm.Console.Sample.dll

Note: If you want to run tests locally via XHarness, you will need to also include /p:JSEngine=NodeJS in that command.

TODO LIST

• Get it to run manually by building and publishing the console sample app for the browser build target and then manually executing the output via node --trace-warnings runtime.js --run Wasm.Console.Sample.dll
• Add a new switch and task to automatically copy over the package.json file and run npm install on it
• Make MSBuild automatically change the emcc Modularize flag based on the new switch
• Add CI build and tests for Node
• Test it!! (Update: Seems to work equally to v8 locally)
• Fix CI build and tests for Node
  • Currently I am disabling the failing tests to get green pipelines so that I can work backwards incrementally to fix the issues later.
• Get and implement feedback

Sample of how it can be used

https://github.com/Daniel-Genkin/runtime/tree/wasm-node-demo/src/mono/sample/wasm/console-node

[ghost]

Tagging subscribers to this area: @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

WORK IN PROGRESS

This PR adds the missing pieces for NodeJS support to MONO WASM!

How does it work

The main differences between this and running the console sample regularly in v8 is that dotnet.js is modularized so we use require to load it. We now also have a package.json and package-lock.json files as well as the node_modules folder. Besides that, most changes to the code were related to various NodeJS related nuances such as the scoping of this but are ignored by the browser.

These changes are triggered by specifying /p:ForNode=True when building and publishing. If parameter is not specified then it is false.

How to run locally

1. ./build.cmd mono+libs -os Browser -c Debug /p:ForNode=True
2. ./dotnet.cmd publish /p:TargetArchitecture=wasm /p:TargetOS=Browser /p:ForNode=True .\src\mono\sample\wasm\console\Wasm.Console.Sample.csproj -c Debug
3. cd src\mono\sample\wasm\console\bin\Debug\AppBundle
4. Run via npm test, ./run-node.sh (or run-node.cmd if on windows) or node runtime.js --run Wasm.Console.Sample.dll

TODO LIST

• Get it to run manually by building and publishing the console sample app for the browser build target and then manually executing the output via node --trace-warnings runtime.js --run Wasm.Console.Sample.dll
• Add a new flag to automatically copy over the package.json file and run npm install on it
• Make MSBuild automatically change the emcc Modularize flag based on flag
• Merge previous PRs ([WASM] Converted mono-config.js to mono-config.json #53606, [WASM] Enabled strict mode on all JS files #54011, [WASM] Normalize runtime-test.js #54281 (reason for runtime-test.js having such a huge diff)) to cleanup the code
• Add a CI build for Node (Should this be a separate PR?)
• Test it!!

Author:	Daniel-Genkin
Assignees:	-
Labels:	area-VM-meta-mono
Milestone:	-

[Daniel-Genkin]

For organizational purposes here is a list of the main test failures(I'll update as I progress):

• (Disabled) JIT tests have issues with npm intermittently not finding a necessary lock file

• (Commented out until Ankit's PR to fix them on Windows lands) Wasm.Build.Tests are failing since the build is caching parameters between runs but Node requires a slightly separate build from v8 and chrome. Note that this is the pre-existing pipeline whereas the others are the new NodeJS ones.

[Daniel-Genkin]

Most of the CI issues right now are caused by the issue that the runfoapp bot reported (#55449)

[thaystg]

/azp run runtime

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Daniel-Genkin]

I just found where the NodeJS and NPM are set for the build machines. Turns out that the coredeps folder (https://github.com/Daniel-Genkin/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/18.04/coredeps/Dockerfile) is a set of dependencies/docker tasks that all of the configurations use. So Helix for example, probably gets it from there. So when we re-enable the currently disabled tests, we can just run npm i -g npm@latest and same for nodejs to update them to the latest versions. However this should probably be a follow up PR

[Daniel-Genkin]

@terrajobst I am an intern on @lewing 's team. So I am internal to MSFT :).

[terrajobst]

@terrajobst I am an intern on @lewing 's team. So I am internal to MSFT :).

Sorry, you didn't show up as an FTE on GitHub. No worries :-)

[Daniel-Genkin]

The current test failures are caused by having an old version of Node on Helix which don't support the newly added FinalizationRegtistry and WeakRef modules which Katelyn added in a recently landed PR. So I am going to update the version. This will also (hopefully) let me re-enable all the tests that I disabled.

Update: As part of my testing for this I found that the minimum supported version of Node is the current LTS (v14)

[Daniel-Genkin]

The current idea is if we need to use node (i.e. ForNode == true) then we will send the emsdk to helix and then use node from inside of it.

Originally the idea (attempt 1) was to download from https://nodejs.org/dist/latest-v14.x/node-v14.17.4-linux-x64.tar.xz but the download system only supports zip files. So using emsdk should get around this while having the added benefit of also keeping the version of node synced with that of emdsk. Maybe after this works we can find a way to strip out node from the emsdk directory instead of sending the whole thing.

[ghost]

/azp run runtime

[azure-pipelines]

Commenter does not have sufficient privileges for PR 54640 in repo dotnet/runtime

[radical]

/azp run runtime

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[pavelsavara]

src/libraries/sendtohelixhelp.proj(396,5): error MSB3923: Failed to download file "https://nodejs.org/dist/latest-v14.x/node-v14.17.4-linux-x64.tar.xz". Response status code does not indicate success: 404 (Not Found).

[pavelsavara]

I'm thinking that the issues with old NodeJs and FinalizationRegtistry/WeakRef could be now OK, as we made them optional.
It's also good test that making it optional actually works.

[pavelsavara]

This PR is superseded by #61313 and #62292