JIT: small OSR locals must be normalize on load by AndyAyersMS · Pull Request #84000 · dotnet/runtime

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

AndyAyersMS · 2023-03-27T22:59:40Z

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load if they were exposed at Tier0.

Fixes#83959.

When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost · 2023-03-27T22:59:55Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

When I changed the importation strategy for OSR in #83910 it exposed a latent issue -- small OSR locals must normalized on load.

Fixes #83959.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2023-03-27T22:59:56Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib

AndyAyersMS · 2023-03-27T23:00:49Z

Expecting a modest number of diffs for OSR methods.

AndyAyersMS · 2023-03-27T23:01:25Z

/azp run runtime-coreclr libraries-pgo

azure-pipelines · 2023-03-27T23:01:36Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-27T23:03:15Z

May also fix #83960 -- checking that now.

AndyAyersMS · 2023-03-27T23:12:29Z

/azp run runtime-coreclr pgostress

azure-pipelines · 2023-03-27T23:12:41Z

Azure Pipelines successfully started running 1 pipeline(s).

AndyAyersMS · 2023-03-28T01:39:42Z

Running into various issues:

Assert failure(PID 24585 [0x00006009], Thread: 24602 [0x601a]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
;; osx
./RunTests.sh: line 168: 50069 Illegal instruction: 4 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B38E099C/w/ACDA0999/e
----- end Mon Mar 27 20:14:10 EDT 2023 ----- exit code 132 ----------------------------------------------------------
exit code 132 means SIGILL Illegal Instruction. Core dumped. Likely codegen issue.
Assert failure(PID 12116 [0x00002f54], Thread: 4788 [0x12b4]): Assertion failed 'op1->gtEffectiveVal() == base' in 'System.RuntimeType:GetMethodBase(System.RuntimeType,int):System.Reflection.MethodBase' during 'Assertion prop' (IL size 480; hash 0xbca2468f; Tier1)
File: D:\a\_work\1\s\src\coreclr\jit\gentree.cpp Line: 4474

AndyAyersMS · 2023-03-28T02:42:13Z

May also fix #83960 -- checking that now.

Seems like it does, but locally I am hitting another odd error. Running on a Coffee Lake i7 8700 so no AVX-512, I end up here when running under WSL2 when built against ed3721b.

cc @tannergooding

## JIT/opt/Compares/compares/compares.sh * thread #1, name = 'corerun', stop reason = signal SIGILL: illegal instruction operand
frame #0: 0x00007ffff776560d libcoreclr.so`RtlRestoreContext at context2.S:140
137 // See https://github.com/apple/darwin-xnu/blob/main/osfmk/i386/fpu.c#L174
138
139 // Restore the ZMM_Hi256 state
-> 140 vinsertf64x4 zmm0, zmm0, ymmword ptr [rdi + (CONTEXT_Zmm0H + 0 * 32)], 1
141 vinsertf64x4 zmm1, zmm1, ymmword ptr [rdi + (CONTEXT_Zmm0H + 1 * 32)], 1
142 vinsertf64x4 zmm2, zmm2, ymmword ptr [rdi + (CONTEXT_Zmm0H + 2 * 32)], 1
143 vinsertf64x4 zmm3, zmm3, ymmword ptr [rdi + (CONTEXT_Zmm0H + 3 * 32)], 1

tannergooding · 2023-03-28T02:53:58Z

This would be caused by #83784

We're supposed to skip this bit if CONTEXT.XStateFeaturesMask hasn't been marked with XSTATE_MASK_AVX512: https://github.com/dotnet/runtime/pull/83784/files#diff-7fb8fda5dbdf85daa35acb2847f3613b2ca50e1dafafae5475e5443589258eacR128-R129

That correspondingly should only be set if FPREG_HasAvx512Registers returns true: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR964-R979

Which itself should only be getting set if the Linux kernel itself reports XFEATURE_MASK_AVX512 in the native thread context: https://github.com/dotnet/runtime/pull/83784/files#diff-6bd99da7ca73739fa65674c40dcb32ecc9af9bcaaff94fdc3d0f4e13704c2331R511-R525

and correspondingly the CPUID query also reports all 5 ISAs as supported: https://github.com/dotnet/runtime/pull/83784/files#diff-eb14b2a3e90e2d0548c465157fcf8480b33fa3b83c1594c068486fe894d32bfeR336-R383

Will need to dig into this a bit to try and repro things...

tannergooding · 2023-03-28T06:46:16Z

Found the issue: #84012

jakobbotsch · 2023-03-28T07:51:23Z

+ (lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);
 }

 bool lvNormalizeOnStore() const
 {
 return varTypeIsSmall(TypeGet()) &&
 // lvIsStructField is treated the same as the aliased local, see fgDoNormalizeOnStore.
- !(lvIsParam || m_addrExposed || lvIsStructField);
+ !(lvIsParam || m_addrExposed || lvIsStructField || lvIsOSRLocal);


It took me a while to guess that the reason is that the local could have been marked normalize-on-load in tier-0 due to the more conservative address exposure there. Maybe add a comment to that effect?
Also, I guess if we wanted to we could scope this down to just OSR locals that were address exposed in tier 0, since we have that information available.

I thought about scoping it down, so let me do that (will need a new bit on lclvar dsc).

AndyAyersMS · 2023-03-28T15:24:10Z

/azp run runtime-coreclr libraries-pgo, runtime-coreclr pgostress

azure-pipelines · 2023-03-28T15:24:28Z

Azure Pipelines successfully started running 2 pipeline(s).

AndyAyersMS · 2023-03-28T17:11:30Z

So far just this one failure has recurred:

 Discovering: System.Runtime.Intrinsics.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Runtime.Intrinsics.Tests (found 1023 test cases)
Starting: System.Runtime.Intrinsics.Tests (parallel test collections = on, max threads = 2)
Assert failure(PID 23193 [0x00005a99], Thread: 23208 [0x5aa8]): Assertion failed '!CodeGen::instIsFP(ins) && (EA_SIZE(attr) <= EA_32BYTE) && (reg != REG_NA)' in 'System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this' during 'Generate code' (IL size 296; hash 0xe5239cf9; Tier1-OSR)
File: /__w/1/s/src/coreclr/jit/emitxarch.cpp Line: 7810
Image: /datadisks/disk1/work/B9AF0A02/p/dotnet

Does not repro locally, suspect it requires AVX-512.

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

AndyAyersMS · 2023-03-28T18:28:32Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitRandomGuardedDevirtualization=1
set DOTNET_JitRandomEdgeCounts=1
set DOTNET_JitRandomlyCollect64BitCounts=1
17:06:48.864 Running test: baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Return code: 1
Raw output file: C:\h\w\BAA209F1\w\ACE309A2\uploads\regressions\2164\foreground-shutdown\output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\BAA209F1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual: 101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED

AndyAyersMS · 2023-03-28T20:25:15Z

@tannergooding how can I repro this w/o having AVX-512 capable hardware?

Bruce pointed out a mismatched VM altjit would work -- I tried this via windows crossjit on linux -- and while I do indeed see zmm's in the disasm, I can't yet repro the assertion failure.

; Assembly listing for method System.Runtime.Intrinsics.Tests.Vectors.Vector512Tests:Vector512SByteShuffleOneInputWithLocalIndicesTest():this
; Emitting BLENDED_CODE for X64 CPU with AVX512 - Windows
; Tier-1 compilation
; OSR variant for entry point 0x11f
...
4889842420010000 mov qword ptr [rsp+120H], rax
62F17C4810B42470030000 vmovups zmm6, zmmword ptr[rsp+370H]
8BB4246C030000 mov esi, dword ptr [rsp+36CH]
...

AndyAyersMS · 2023-03-28T23:25:27Z

Also seeing an arm64 failure. Not sure it is related but will take a look.

There aren't any OSR methods in this test, so the failure seems unrelated.

AndyAyersMS · 2023-03-28T23:26:06Z

Going to merge since we have potential OSR silent bad code; will try and repro the AVX512 issue some other way.

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

JIT: small OSR locals must be normalize on load
cac326f
When I changed the importation strategy for OSR in dotnet#83910 it exposed a latent issue -- small OSR locals must normalized on load. Fixesdotnet#83959.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 27, 2023

ghost assigned AndyAyersMS Mar 27, 2023

AndyAyersMS requested a review from jakobbotsch March 27, 2023 23:00

build-analysisBot mentioned this pull request Mar 28, 2023
CI failure STRICT_JS doesn't work with MODULARIZE or EXPORT_ES6 #83986
Closed

tannergooding mentioned this pull request Mar 28, 2023
Ensure that CONTEXT_XStateFeaturesMask is explicitly zeroed out on RtlCaptureContext #84012
Merged

jakobbotsch reviewed Mar 28, 2023
View reviewed changes

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS added 2 commits March 28, 2023 08:21

Merge branch 'main' into Fix83959
d3e9fcc

feedback; restrict to tier0 exposed locals
b124a16

This was referenced Mar 28, 2023
IOException running NuGet-Migrations during tests in dotnet CLI first run #80619
Closed
[release/6.0] Doublelinklist GC failures on Mono #83245
Closed

jakobbotsch approved these changes Mar 28, 2023
View reviewed changes

AndyAyersMS merged commit fd157a5 into dotnet:mainMar 28, 2023

AndyAyersMS mentioned this pull request Mar 30, 2023
On Stack Replacement Next Steps #33658
Open
72 tasks

ghost locked as resolved and limited conversation to collaborators Apr 28, 2023

Uh oh!

Conversation

AndyAyersMS commented Mar 27, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

azure-pipelinesBot commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

azure-pipelinesBot commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 28, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

tannergooding commented Mar 28, 2023

Uh oh!

tannergooding commented Mar 28, 2023

Uh oh!

jakobbotschMar 28, 2023

Choose a reason for hiding this comment

Uh oh!

AndyAyersMSMar 28, 2023

Choose a reason for hiding this comment

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

azure-pipelinesBot commented Mar 28, 2023

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

AndyAyersMS commented Mar 28, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

AndyAyersMS commented Mar 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AndyAyersMS commented Mar 27, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

azure-pipelinesBot commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023

Uh oh!

azure-pipelinesBot commented Mar 27, 2023

Uh oh!

AndyAyersMS commented Mar 27, 2023•
edited
Loading

AndyAyersMS commented Mar 28, 2023•
edited
Loading

AndyAyersMS commented Mar 28, 2023•
edited
Loading

AndyAyersMS commented Mar 27, 2023•
edited
Loading

AndyAyersMS commented Mar 28, 2023•
edited
Loading

AndyAyersMS commented Mar 28, 2023•
edited
Loading

AndyAyersMS commented Mar 27, 2023•
edited
Loading