Skip to content

Fix flaky SendAsync_Success_ConnectionSetupActivityGraphRecorded test (Http1.1 + Http2) - #129822

Closed
lewing with Copilot wants to merge 3 commits into
mainfrom
copilot/diagnosticstest-fix-unexpected-eof
Closed

Fix flaky SendAsync_Success_ConnectionSetupActivityGraphRecorded test (Http1.1 + Http2)#129822
lewing with Copilot wants to merge 3 commits into
mainfrom
copilot/diagnosticstest-fix-unexpected-eof

Conversation

CopilotAI commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

SendAsync_Success_ConnectionSetupActivityGraphRecorded was intermittently failing with an AggregateException containing both the real assertion failure and a spurious server-side EOF — because the loopback server raced to read req2 before the client finished asserting req1 and sent it. On connection close, Http/1.1 threw "Unexpected EOF trying to read request header" and Http/2 threw "Failed to read Headers frame.".

Fix: Add a TaskCompletionSource to synchronize client and server within the test:

  • Client signals req1AssertionsDone.SetResult() after all req1 assertions pass, immediately before sending req2
  • Server awaits req1AssertionsDone.Task before calling ReadRequestDataAsync() for req2

This eliminates the race for both Http/1.1 and Http/2 variants. If a client assertion fails, the TCS is never signaled, the server waits indefinitely, and WhenAllOrAnyFailed surfaces only the real assertion failure.

CopilotAI requested review from Copilot and removed request for CopilotJune 24, 2026 21:14
@lewinglewing added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-System.Net.Http labels Jun 24, 2026
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

… with TaskCompletionSource sync
Co-authored-by: lewing <24063+lewing@users.noreply.github.com>
CopilotAI requested review from Copilot and removed request for CopilotJune 24, 2026 22:11
CopilotAI changed the title [WIP] Fix unexpected EOF in DiagnosticsTest SendAsync_Success_ConnectionSetupActivityGraphRecordedFix flaky SendAsync_Success_ConnectionSetupActivityGraphRecorded test (Http1.1 + Http2)Jun 24, 2026
CopilotAI requested a review from lewingJune 24, 2026 22:11

@lewinglewing left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is misdiagnosed and shouldn't be merged. The "race between client assertions and server reading req2" framing isn't the cause of the flake — there is no such race. The loopback request/response pump already serializes the two sides.

Actual cause: the JIT regression in #128384 (merged 2026-06-22 23:47 UTC, commit b426fdf). The client-side assertion at DiagnosticsTests.cs:657req1.Links.Single(l => l.Context == conn.Context) — throws "Sequence contains no matching element" because the auto-promoted runtime-async variant of a sync Task/ValueTask-returning forwarder in the System.Net.Http send chain is miscompiled (same class of bug as the deterministic NRE Rolf reported in #129813). The server-side "Unexpected EOF" is purely downstream: once the client Single() throws, WhenAllOrAnyFailed cancels, the loopback socket closes mid-ReadRequestDataAsync, and the EOF gets bundled into the AggregateException.

Evidence this is a runtime regression, not a test race:

This test was added in #103922 (July 2024) and has been clean on rolling-main runtime CI continuously through 2026-06-22. First failure on 2026-06-23 — the first rolling-main build containing #128384:

Build (def 129 main)Finished (UTC)sourceVersionHas #128384?ConnectionSetupActivityGraphRecorded failures
146600406-16 14:1298d5787no0
146945106-18 07:5926243adno0
147213806-19 10:1863ac729no0
147505406-22 11:2859979e6no0
147612606-23 00:0997702a1no0
147679906-23 10:50461fd2ayes12
147812606-24 00:49b5e2befyes12
147886206-24 10:1646b72c7yes12

A test race that lay dormant for ~2 years and started intermittently failing on exactly the commit that introduced auto-promotion of synchronous task-returning methods to runtime-async is not a test race.

The "fix" also makes the failure mode worse when the regression still fires:

  • Today:AggregateException with the real Single() assertion failure + server-side EOF. Ugly, but the real failure is in the message.
  • With this PR: client's Single() throws → req1AssertionsDone.SetResult() is never called → server blocks forever in await req1AssertionsDone.TaskWhenAllOrAnyFailed waits for the server task → test hangs until xunit's per-test timeout instead of failing fast. The PR description claims this as a feature ("the server waits indefinitely, and WhenAllOrAnyFailed surfaces only the real assertion failure"), but WhenAllOrAnyFailed propagates the first failure immediately; a hung server task just delays surfacing the real one.

Recommendation: Close this PR. The fix belongs in src/coreclr/jit/importer.cpp (impWrapTopOfStackInAwait / impSetupAsyncCall) — Jakob is already assigned to #129813 which has a deterministic minimal repro (Stream.ReadAsync(Memory<byte>) NRE) and is the same bug. The diagnostics test should stay as-is so it keeps surfacing the regression until the JIT fix lands; suppressing the secondary EOF noise here would mask the JIT bug across other code paths #128384 has broken.

Note

This comment was authored with assistance from GitHub Copilot.

Co-authored-by: lewing <24063+lewing@users.noreply.github.com>
CopilotAI requested review from Copilot and removed request for CopilotJune 24, 2026 22:20

CopilotAI commented Jun 24, 2026

Copy link
Copy Markdown
ContributorAuthor

This PR is misdiagnosed and shouldn't be merged. The "race between client assertions and server reading req2" framing isn't the cause of the flake — there is no such race. The loopback request/respo...

You're right. The diagnosis was incorrect and the fix makes things worse by turning a fast failure into a hang. I've reverted DiagnosticsTests.cs to its original state (commit ce71db32~1) — the test is back to exactly what it was before this PR.

CopilotAI requested a review from lewingJune 24, 2026 22:20
@lewinglewing closed this Jun 24, 2026
@jakobbotsch

Copy link
Copy Markdown
Member

and is the same bug

I'm doubtful. The other bug is related to stripping of IL bodies and the interpreter trying to execute the stripped bodies. This one looks like x86, I don't think we do anything like that there.
I'll take a look tomorrow.

@jkotas
jkotas deleted the copilot/diagnosticstest-fix-unexpected-eof branch July 3, 2026 18:53
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Aug 3, 2026
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jakobbotsch@lewing