Copy published crossgen2 in artifacts/tests

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.

[am11]

Fixes#80110

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Fixes #80110

Author:	am11
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

Overall looks good to me, thanks Adeel, in general I believe this to be heading in the right direction. We probably need to soften some rough edges though. It seems to me that quite a few R2R tests failed in your PR run, that will require investigation and fixing (theoretically these could be caused by the weird addition of ".." in populating CORE_ROOT I commented on in the review as I find that surprising). Please also see my comment regarding the filtering on checked build configuration. Once we're on the same page w.r.t. these aspects, I'll be more than happy to approve your change and merge it in.

[trylek]

In our today .NET core runtime sync several people emphasized that we should coordinate this work with the diagnostics team to make sure we don't paint ourselves in a corner due to the fact that today NativeAOT bugs are in general harder to analyze and investigate. /cc-ing @tommcdon for visibility.

[am11]

Thank you! BinaryFormatter failure was #104216.

Follow up list is available here: #106929.

[jakobbotsch]

With this PR the repo no longer builds for me. I get this error:

C:\dev\dotnet\runtime2\.dotnet\sdk\9.0.100-preview.7.24407.12\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceRes
olution.targets(495,5): error NETSDK1112: The runtime pack for Microsoft.NETCore.App.Runtime.win-x64 was not downloaded. Try running
a NuGet restore with the RuntimeIdentifier 'win-x64'. [C:\dev\dotnet\runtime2\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csp
roj]

Build command is .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly.

[jakobbotsch]

Also, superpmi-collect fails during "Create Core_Root" step on linux-arm (and probably other platforms) with:

2024-08-25T18:07:53.9782152Z Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 2024-08-25T18:07:53.9787522Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788209Z /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64.so)
2024-08-25T18:07:53.9788573Z libclrjit_universal_arm_x64.so: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9788916Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/clrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790253Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/libclrjit_universal_arm_x64: cannot open shared object file: No such file or directory
2024-08-25T18:07:53.9790466Z 2024-08-25T18:07:54.0025059Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-25T18:07:54.0025730Z at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-25T18:07:54.0026210Z at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-25T18:07:54.0026541Z 2024-08-25T18:07:54.0026748Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027006Z at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-25T18:07:54.0027324Z at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-25T18:07:54.0027706Z at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-25T18:07:54.0028041Z at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-25T18:07:54.0028416Z at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-25T18:07:54.0028690Z at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-25T18:07:54.0316349Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-25T18:07:54.5680688Z Determining projects to restore...
2024-08-25T18:07:55.8919801Z All projects are up-to-date for restore.
2024-08-25T18:07:55.9387359Z 2024-08-25T18:07:55.9388368Z Build FAILED.

Edit: Looks like linux-arm and linux-arm64 are broken while other targets are ok.

Edit 2: I'm guessing the problem is somewhere in the following that does not end up producing the right crossgen2 bits:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 112 to 151 in 5fd3f22

	- template: /eng/pipelines/common/platform-matrix.yml
	parameters:
	jobTemplate: /eng/pipelines/common/global-build-job.yml
	buildConfig: checked
	platforms:
	- linux_arm
	- linux_arm64
	jobParameters:
	testGroup: outerloop
	buildArgs: -s clr+libs+libs.tests -rc $(_BuildConfig) -c Release /p:ArchiveTests=true
	timeoutInMinutes: 120
	postBuildSteps:
	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64
	- template: /eng/pipelines/coreclr/templates/build-native-test-assets-step.yml
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/bin
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	- template: /eng/pipelines/common/upload-artifact-step.yml
	parameters:
	rootFolder: $(Build.SourcesDirectory)/artifacts/helix
	includeRootFolder: false
	archiveType: $(archiveType)
	archiveExtension: $(archiveExtension)
	tarCompression: $(tarCompression)
	artifactName: LibrariesTestArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)
	extraVariablesTemplates:
	- template: /eng/pipelines/common/templates/runtimes/native-test-assets-variables.yml
	parameters:
	testGroup: outerloop
	disableComponentGovernance: true # No shipping artifacts produced by this pipeline

[jakobbotsch]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

[am11]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=787949&view=results also appears to have some failures that look related in the "Test crossgen2-comparison X to Y" jobs.

Can you pinpoint the error? Those failures look to be same as #106948.

[am11]

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been):
.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

[jakobbotsch]

Can you pinpoint the error? Those failures look to be same as #106948.

For example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-600889e483e24676bf/WorkItem/1/console.51e0e9bf.log?helixlogtype=result

.\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly

The supported command is (and always have been): .\build.cmd clr+libs -rc checked -c release && .\src\tests\build.cmd checked generatelayoutonly -p:LibrariesConfiguration=Release

I have built this way for 3 years without issues. I do not see a mention that LibrariesConfiguration needs to be specified always in the workflow docs (actually I see the opposite). https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

[am11]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

[jakobbotsch]

From main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

Note that for the libraries configuration, we are passing the argument directly to MSBuild instead of the build script, hence the /p:LibrariesConfiguration flag. Also, make sure you use the correct syntax depending on our platform. The cmd script takes the arguments by placing, while the sh script requires them to be with a hyphen.

Yes, for debug libraries I understand that it needs to be specified. I never build debug libraries, so I never have needed to specify it -- section below what you quote mentions that it is only necessary if your libraries configuration is not release.

[am11]

The fix is simple (set LibrariesConfiguration = Release by default). We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

[jakobbotsch]

The fix is simple (set LibrariesConfiguration = Release by default).

👍

We did ran couple of pipelines, including superpmi by making a dummy change. Apparently it wasn't enough.

In this case superpmi-collect is the pipeline that substantially uses crossgen2, while superpmi-diffs/superpmi-replay do not use it at all. Those latter pipelines need only JIT + SPMI builds and then download the collections created by superpmi-collect.
Sadly superpmi-collect is not public, so we (the JIT team) will have to help a bit in getting this working.

One thing that confuses me a bit is that it seems we are running ilc even when generatelayoutonly is specified. I would not expect generatelayoutonly to start running ILC and prejitting things.

/azp run runtime-coreclr crossgen2

We were using that as an ultimate test and didn't ran crossgen2 outerloop.

To be fair I am not sure that the crossgen2 outerloop pipeline is checked much. I just noticed it while trying to find a publicly accessible pipeline run that hit the same issue as superpmi-collect (since it is only internally accessible).

[jakobbotsch]

Sadly superpmi-collect is not public

I think we can add a public test-only version of this pipeline that does everything except upload the collections. That should make the testing much more straightforward for contributors.

[am11]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

[jakobbotsch]

If it is blocking, lets revert it. I think two of the fixes are striaghtforward (add a line in src/tests/Common/Directory.Build.props and update crosgen2 pipeline yamls to drop .dll).

Yup, the superpmi-collect issue needs some investigation. It maybe as simple as passing --cross to:

runtime/eng/pipelines/coreclr/superpmi-collect.yml

Lines 124 to 129 in 5fd3f22

	# Build CLR assets for x64 as well as the target as we need an x64 mcs
	- template: /eng/pipelines/common/templates/global-build-step.yml
	parameters:
	buildArgs: -s clr.spmi -c $(_BuildConfig)
	archParameter: -arch x64
	container: linux_x64

or something more involved. I'll take a closer look when I am off my day job. 😅

Thanks! I think we should revert it for now as otherwise we'll quickly end up with no SPMI collections for linux-arm/linux-arm64 while we investigate.