Preserve last error for patchpoint helpers by AndyAyersMS · Pull Request #75922 · dotnet/runtime

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

AndyAyersMS · 2022-09-20T19:29:10Z

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes#75828.

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error. Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this. Fixesdotnet#75828.

ghost · 2022-09-20T19:29:25Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

In stress modes (and eventually perhaps in normal uses) the jit may insert patchpoint helper calls in regions where last error is live. So the helpers need to preserve last error.

Because some invocations of the helpers may transition to OSR methods instead of returning, we can't use the normal macros for this.

Fixes #75828.

Author:	AndyAyersMS
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

AndyAyersMS · 2022-09-20T19:30:08Z

@jakobbotsch PTAL
cc @dotnet/jit-contrib @jkotas

Seems like EnC could hit this issue too?

jkotas · 2022-09-20T19:34:43Z

 void JIT_Patchpoint(int* counter, int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:35:45Z

 void JIT_PartialCompilationPatchpoint(int ilOffset)
 {
+ // BEGIN_PRESERVE_LAST_ERROR;
+ DWORD __dwLastError = ::GetLastError();


Suggested change
DWORD__dwLastError = ::GetLastError();
DWORDdwLastError = ::GetLastError();

jkotas · 2022-09-20T19:40:42Z

Seems like EnC could hit this issue too?

I do not see any obvious problem. EnC patch runs inside debugger step handler that is preserving last error:

runtime/src/coreclr/vm/exceptionhandling.cpp

Line 857 in 5384105

DWORD dwLastError = GetLastError();

AndyAyersMS · 2022-09-20T21:14:11Z

OSX seems to have some networking issues:

fatal: unable to access 'https://github.com/dotnet/runtime/': The requested URL returned error: 429
##[warning]Git fetch failed with exit code 128, back off 8.881 seconds before retry.

stephentoub · 2022-09-20T21:17:23Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

MattGal · 2022-09-20T21:38:39Z

Yeah, almost every PR today has failed most mac legs because of that. @MattGal is investigating.

The "normal" channels are being a bit slow but I have confirmed they're aware of the throttling problem and are trying to identify the cause presently.

jkotas · 2022-09-21T13:19:25Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

AndyAyersMS · 2022-09-21T15:25:16Z

Is this .NET 7 backport candidate? (Intermittent data corruption caused by OSR.)

I don't know if we can hit this with normal OSR.

The failure above is from a stress mode that places patchpoints in randomly selected blocks where the stack is empty, whereas OSR normally will only place patchpoints at blocks that are the start or end of a loop.

I'll spend some time today trying to work up a case; seems like the "end of loop" patchpoint (which is the default placement) might be a place we could hit this normally.

jkotas · 2022-09-21T15:30:20Z

I would expect that something like this can hit this reliably:

int test()
{
Marshal.SetLastSystemError(42);
int fact = 1;
for (int i = 0; i < 2_000_000_000; i++)
{
fact *= i;
}
return fact + Marshal.GetLastSystemError();
}

AndyAyersMS · 2022-09-21T18:03:19Z

I would expect that something like this can hit this reliably:

Yes, thanks.

Repro is simple and deterministic; the failure happens in Tier0 code once it has invoked the patchpoint helper. Will add a test case.

AndyAyersMS · 2022-09-21T20:11:13Z

For Linxu arm64 NAOT: msbuild crashed during restore:

/__w/1/s/.dotnet/sdk/7.0.100-preview.7.22377.5/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22469.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 474: 535 Segmentation fault (core dumped) "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.
Finishing: Build product

Similar perhaps to #43826 but that issue is quite old.

AndyAyersMS · 2022-09-22T01:37:58Z

@jeffschwMSFT @JulieLeeMSFT we should consider porting this to 7.0.

jeffschwMSFT · 2022-09-22T14:06:15Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

AndyAyersMS · 2022-09-22T15:00:46Z

@AndyAyersMS in what cases (beyond stress modes) do we anticipate that this may occur?

This PR has a simple test case that fails without stress -- the value of last error will not be preserved across loops.

jkotas · 2022-09-22T15:04:28Z

In a more real-world situation, the long-running for-loop in the test can be some kind of interop marshalling code.

AndyAyersMS · 2022-09-26T18:31:25Z

/backport to release/7.0

github-actions · 2022-09-26T18:31:36Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3130148038

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 20, 2022

ghost assigned AndyAyersMS Sep 20, 2022

jkotas reviewed Sep 20, 2022
View reviewed changes

jkotas approved these changes Sep 20, 2022
View reviewed changes

fix build issues, review feedback
4cc615b

jakobbotsch approved these changes Sep 21, 2022
View reviewed changes

add test case
05957af

AndyAyersMS merged commit c81e7a2 into dotnet:mainSep 22, 2022

This was referenced Sep 22, 2022
msbuild segfault on arm64 linux package restore #76039
Closed
On Stack Replacement Next Steps #33658
Open

github-actionsBot mentioned this pull request Sep 26, 2022
[release/7.0] Preserve last error for patchpoint helpers #76200
Merged

ghost locked as resolved and limited conversation to collaborators Oct 26, 2022

	DWORD__dwLastError = ::GetLastError();
	DWORDdwLastError = ::GetLastError();

Uh oh!

Conversation

AndyAyersMS commented Sep 20, 2022

Uh oh!

ghost commented Sep 20, 2022

Uh oh!

AndyAyersMS commented Sep 20, 2022

Uh oh!

jkotasSep 20, 2022

Choose a reason for hiding this comment

Uh oh!

jkotasSep 20, 2022

Choose a reason for hiding this comment

Uh oh!

jkotas commented Sep 20, 2022

Uh oh!

AndyAyersMS commented Sep 20, 2022

Uh oh!

stephentoub commented Sep 20, 2022

Uh oh!

MattGal commented Sep 20, 2022

Uh oh!

jkotas commented Sep 21, 2022

Uh oh!

AndyAyersMS commented Sep 21, 2022

Uh oh!

jkotas commented Sep 21, 2022

Uh oh!

AndyAyersMS commented Sep 21, 2022

Uh oh!

AndyAyersMS commented Sep 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndyAyersMS commented Sep 22, 2022

Uh oh!

jeffschwMSFT commented Sep 22, 2022

Uh oh!

AndyAyersMS commented Sep 22, 2022

Uh oh!

jkotas commented Sep 22, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndyAyersMS commented Sep 26, 2022

Uh oh!

github-actionsBot commented Sep 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

AndyAyersMS commented Sep 20, 2022

Uh oh!

ghost commented Sep 20, 2022

Uh oh!

AndyAyersMS commented Sep 20, 2022

Uh oh!

jkotasSep 20, 2022

Choose a reason for hiding this comment

Uh oh!

jkotasSep 20, 2022

Choose a reason for hiding this comment

Uh oh!

jkotas commented Sep 20, 2022

Uh oh!

AndyAyersMS commented Sep 20, 2022

Uh oh!

stephentoub commented Sep 20, 2022

Uh oh!

MattGal commented Sep 20, 2022

Uh oh!

jkotas commented Sep 21, 2022

Uh oh!

AndyAyersMS commented Sep 21, 2022

Uh oh!

AndyAyersMS commented Sep 21, 2022•
edited
Loading

jkotas commented Sep 22, 2022•
edited
Loading

AndyAyersMS commented Sep 21, 2022•
edited
Loading

jkotas commented Sep 22, 2022•
edited
Loading