Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)"

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.

[am11]

Reverts #106956

[am11]

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

[jakobbotsch]

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

/azp run Fuzzlyn

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jakobbotsch]

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948#106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task : Command line
2024-08-26T15:07:42.7583063Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version : 2.237.1
2024-08-26T15:07:42.7583197Z Author : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh" --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 2024-08-26T15:07:55.3341936Z [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z 2024-08-26T15:07:56.2277389Z Determining projects to restore...
2024-08-26T15:08:02.8336187Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z 2024-08-26T15:08:03.8483224Z Determining projects to restore...
2024-08-26T15:08:05.0716183Z Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z 2024-08-26T15:08:06.0119968Z Determining projects to restore...
2024-08-26T15:08:07.6024028Z Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z 2024-08-26T15:08:08.5017637Z Determining projects to restore...
2024-08-26T15:08:09.2113281Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z 2024-08-26T15:08:10.1003679Z Determining projects to restore...
2024-08-26T15:08:10.8072973Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z 2024-08-26T15:08:11.7144653Z Determining projects to restore...
2024-08-26T15:08:12.4401917Z Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z 2024-08-26T15:08:13.3232739Z Determining projects to restore...
2024-08-26T15:08:14.0382600Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z 2024-08-26T15:08:14.9241455Z Determining projects to restore...
2024-08-26T15:08:18.6261668Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z 2024-08-26T15:08:19.5546027Z Determining projects to restore...
2024-08-26T15:08:20.2806869Z Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z 2024-08-26T15:08:21.1573576Z Determining projects to restore...
2024-08-26T15:08:22.5306653Z Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z 1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z 2024-08-26T15:08:23.5122215Z Determining projects to restore...
2024-08-26T15:08:25.3625703Z Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z 2024-08-26T15:08:28.5176711Z Determining projects to restore...
2024-08-26T15:08:29.7457355Z Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z Building Targeting Pack
2024-08-26T15:08:30.4660518Z external -> 2024-08-26T15:08:31.0923175Z Determining projects to restore...
2024-08-26T15:08:33.3667587Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z Generating native code
2024-08-26T15:09:43.3319288Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-26T15:09:43.3320904Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z 2024-08-26T15:09:43.3526129Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z 2024-08-26T15:09:43.3530523Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z Determining projects to restore...
2024-08-26T15:09:45.0851704Z All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z 0 Warning(s)
2024-08-26T15:09:45.1225288Z 1 Error(s)
2024-08-26T15:09:45.1225694Z 2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

[am11]

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

[am11]

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll
" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp" --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll
" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" CoreMangLib.dll Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false
CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
Image: C:\h\w\BAE10A27\p\corerun.exe
App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

[am11]

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

[am11]

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

I think I have figured it out. Please run superpmi-collect and crossgen2 outerloop to see if it goes through. Also fixed the LibrariesConfiguration=Release default in the third commit.

[am11]

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

[am11]

I hope now this is ready. Lets run the legs again to see it in action. 😎

[jkotas]

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

[jakobbotsch]

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

[am11]

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

[jkotas]

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

• Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
• Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
• Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

[jakobbotsch]

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jakobbotsch]

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

[am11]

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

[jakobbotsch]

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2 -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log [15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)
C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe ERROR: The process "corerun.exe" not found.
C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 ['0' END OF WORK ITEM LOG: Command exited with 1]

[am11]

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

[jakobbotsch]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[am11]

Remaining failures are same as the one in main. Opened #107226.

[jakobbotsch]

Thanks for fixing the remaining problems.