[wasm] R2R OSR pinnedlocal hits fFrameWasUnwound and cycles while dumping frames

[lewing]

Description

When the browser-wasm CoreCLR JIT/opt/OSR/pinnedlocal test is crossgened and run as ReadyToRun code, its out-of-process preflight asserts in StackFrameIterator::Filter at stackwalk.cpp:1721 because fFrameWasUnwound is false. The subsequent diagnostic frame dump repeatedly reports the same InlinedCallFrame until the existing 300-second watchdog terminates the test.

This was observed while validating #133276 at commit bec55a87be6f2f77518cc9316e57330454b98897. It is being tracked separately because an exact R2R baseline control has not yet established whether that change introduced the failure. The interpreter control passes this test.

Reproduction Steps

1. Check out dotnet/runtime with Add preemptive-only cooperative GC transition holder #133276 at bec55a87be6f2f77518cc9316e57330454b98897.
2. Build browser-wasm Checked CoreCLR and generate a crossgen2 layout:

export PATH="$PWD/artifacts/pyshim:$PWD/.dotnet:$PATH"
./build.sh clr -os browser -rc checked
./src/tests/build.sh generatelayoutonly crossgen2 checked \
 -arch wasm -os browser -p:HostConfiguration=Release

3. Clear stale JIT/opt IL-CG2 images and run the generated JIT.opt wrapper with ReadyToRun enabled:

export CORE_ROOT="$PWD/artifacts/tests/coreclr/browser.wasm.Checked/Tests/Core_Root"
RunCrossGen2=1 bash JIT.opt.sh -coreroot="$CORE_ROOT"

The failure occurs during the out-of-process preflight for JIT/opt/OSR/pinnedlocal.

Expected behavior

JIT/opt/OSR/pinnedlocal completes successfully under browser-wasm ReadyToRun, as it does with ReadyToRun disabled.

Actual behavior

The test asserts at src/coreclr/vm/stackwalk.cpp:1721:

fFrameWasUnwound

The diagnostic dump then repeatedly emits the same InlinedCallFrame and does not terminate normally. The existing 300-second watchdog eventually ends the test.

Regression?

Unknown. The failure was observed on #133276, but no exact baseline R2R control has been run, so it should not be attributed to that PR. The interpreter control passes.

Known Workarounds

Run the test with ReadyToRun disabled (DOTNET_ReadyToRun=0). The test passes in that configuration.

Configuration

• CoreCLR browser-wasm
• Checked runtime
• ReadyToRun image generated with crossgen2
• macOS ARM64 build host
• Node.js 23.11.1
• Test sets DOTNET_GCStress=3

Other information

This is distinct from:

• Assert failure: OBJECTREF accessed in preemptive GC mode in browser-wasm ReadyToRun runtime tests #133219 and the cooperative GC-mode assertion seen in other wasm R2R tests;
• [wasm] IsManagedCode misclassifies R2R virtual IPs during funclet stack walks #133264 / [wasm] Recognize R2R virtual IPs in IsManagedCode #133266, which address wasm virtual-IP classification during funclet stack walks.

The observed cyclic InlinedCallFrame dump suggests the frame chain or unwind state should be investigated at the first fFrameWasUnwound assertion rather than treating the watchdog termination as the root cause.

Note

This issue was created with GitHub Copilot.

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/crossgen-contrib
See info in area-owners.md if you want to be subscribed.