Remove boxing for awaited custom awaiters

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330

[jakobbotsch]

This optimizes calls to AsyncHelpers.UnsafeAwaitAwaiter<TAwaiter>(TAwaiter) with struct awaiters to instead call a new function AsyncHelpers.UnsafeAwaitAwaiterInContinuation<TAwaiter>(int offset). The idea is that the JIT ensures that the awaiter will be present in the continuation at the specified offset. The later UnsafeOnCompleted call can then be done without any boxing by extracting it from the continuation.

This introduces a new GT_CONTINUATION_MEMBER_OFFSET which is used to solve the linking problem where we do not know the offset into the continuation until much later. The async transformation is responsible for replacing this node with a constant after it knows the offset.
There is a new AsyncAwaiter pseudo arg passed to UnsafeAwaitAwaiterInContinuation, and expanded in the suspension path by the async transformation to be stored in the continuation at the right offset.

I have a couple of use cases for GT_CONTINUATION_MEMBER_OFFSET in mind, so I have made the mechanism to represent the type of member easily expandable (particularly inlining needs this as well).

This PR also removes configurable continuation reuse. This is now unconditionally enabled, to simplify the expansion of GT_CONTINUATION_MEMBER_OFFSET nodes.

Fix#119842

Microbenchmark

About 10% improvement, and more importantly, avoids the box that async1 also avoids to gain parity.

usingSystem;usingSystem.Diagnostics;usingSystem.Runtime.CompilerServices;usingSystem.Threading;usingSystem.Threading.Tasks;publicstaticclassProgram{staticActions_continuation;staticlongs_value;publicstaticvoidMain(){for(inti=0;i<10;i++){for(intj=0;j<100;j++){Taskt=Foo(100);while(!t.IsCompleted)s_continuation();}Thread.Sleep(100);}for(inti=0;i<50;i++){Taskt=Foo(10_000_000);while(!t.IsCompleted)s_continuation();}}privatestaticasyncTaskFoo(intn){s_value=0;Stopwatchtimer=Stopwatch.StartNew();for(inti=0;i<n;i++){awaitnewAwaiter(i);}if(n>100)Console.WriteLine("Took {0} ms",timer.ElapsedMilliseconds);Trace.Assert(s_value==((long)n*(n-1))/2);}privatestructAwaiter:ICriticalNotifyCompletion{publicintX;publicAwaiter(intx)=>X=x;publicboolIsCompleted=>false;publicAwaiterGetAwaiter()=>this;publicvoidGetResult(){}publicvoidOnCompleted(Actioncontinuation){}publicvoidUnsafeOnCompleted(Actioncontinuation){s_value+=X;s_continuation=continuation;}}}

-Took 323 ms-Took 318 ms-Took 320 ms-Took 323 ms-Took 322 ms-Took 318 ms-Took 320 ms-Took 318 ms-Took 317 ms-Took 319 ms+Took 290 ms+Took 291 ms+Took 292 ms+Took 288 ms+Took 292 ms+Took 291 ms+Took 290 ms+Took 290 ms+Took 287 ms+Took 292 ms

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR extends CoreCLR runtime-async to avoid boxing custom struct awaiters on suspension by storing the awaiter value into the continuation object and later invoking OnCompleted/UnsafeOnCompleted by extracting the awaiter from that continuation using a JIT-computed byte offset. It introduces a symbolic “continuation member offset” node so the importer can reference the eventual offset before the async transformation finalizes the continuation layout, and adds a new JIT↔EE query to resolve the specialized helper method.

Changes:

• Add new CoreLib helpers to await via “awaiter-in-continuation” with an offset, and update suspension handling to call OnCompleted/UnsafeOnCompleted by reading the awaiter out of the continuation.
• Add GT_CONTINUATION_MEMBER_OFFSET plus continuation-member layout plumbing in the JIT async transformation; update importer to rewrite struct-await helper calls to the new helper and record the member offset.
• Extend the JIT↔EE interface and supporting tooling (VM, SuperPMI, ILC/ReadyToRun) to resolve the new helper method; add a regression test covering safe/unsafe custom awaiters.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tests/async/struct/struct.cs	Adds a regression test for safe and unsafe custom struct awaiters under runtime async.
src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs	Adds new private awaiter-in-continuation helpers and extraction-based OnCompleted/UnsafeOnCompleted paths.
src/coreclr/vm/jitinterface.h	Adds VM declaration for the new JIT↔EE query to resolve the awaiter-in-continuation helper.
src/coreclr/vm/jitinterface.cpp	Implements helper resolution + runtime lookup computation for the new awaiter-in-continuation call.
src/coreclr/vm/corelib.h	Adds CoreLibBinder method IDs for the new AsyncHelpers helper methods.
src/coreclr/tools/superpmi/superpmi/icorjitinfo.cpp	Plumbs the new ICorJitInfo method through SuperPMI.
src/coreclr/tools/superpmi/superpmi-shim-simple/icorjitinfo_generated.cpp	Updates generated shim to forward the new ICorJitInfo entrypoint.
src/coreclr/tools/superpmi/superpmi-shim-counter/icorjitinfo_generated.cpp	Updates generated counter shim to track/forward the new call.
src/coreclr/tools/superpmi/superpmi-shim-collector/icorjitinfo.cpp	Updates collector shim to record/replay the new query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.h	Adds recording/replay declarations and a new packet kind for the query.
src/coreclr/tools/superpmi/superpmi-shared/methodcontext.cpp	Implements record/dump/replay for the new query.
src/coreclr/tools/superpmi/superpmi-shared/lwmlist.h	Adds LWM map entry for the query.
src/coreclr/tools/superpmi/superpmi-shared/agnostic.h	Adds agnostic key struct for recording the query inputs.
src/coreclr/tools/Common/JitInterface/ThunkGenerator/ThunkInput.txt	Adds the new API to the JIT interface thunk generator input.
src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs	Adds ILC implementation to resolve the new helper method handle.
src/coreclr/tools/Common/JitInterface/CorInfoImpl_generated.cs	Adds generated unmanaged callback plumbing for the new API.
src/coreclr/tools/aot/jitinterface/jitinterface_generated.h	Updates NativeAOT jitinterface wrapper with the new callback.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs	Ensures R2R token availability for the new helper methods.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs	Adds runtime stack-state fields + dispatcher logic to call awaiter continuation callbacks.
src/coreclr/jit/wellknownargs.h	Introduces a new well-known arg kind for the async awaiter pseudo-arg.
src/coreclr/jit/importercalls.cpp	Adds importer rewrite to swap struct awaiter calls to the new helper and attach symbolic continuation-member offset.
src/coreclr/jit/ICorJitInfo_wrapper_generated.hpp	Adds wrapper method for the new JIT↔EE query.
src/coreclr/jit/ICorJitInfo_names_generated.h	Adds the new API name for logging/tracing.
src/coreclr/jit/gtstructs.h	Extends GenTreeVal struct mapping to include GT_CONTINUATION_MEMBER_OFFSET.
src/coreclr/jit/gtlist.h	Adds the GT_CONTINUATION_MEMBER_OFFSET node kind and documentation.
src/coreclr/jit/gentree.cpp	Updates pseudo-arg handling and debug printing for the new node.
src/coreclr/jit/compiler.h	Adds compiler state for tracking continuation members and the importer helper prototype.
src/coreclr/jit/async.h	Adds continuation-member abstractions and extends layout builder to allocate member offsets.
src/coreclr/jit/async.cpp	Implements continuation-member tracking, reserves storage in continuation layouts, and bashes symbolic offsets to constants.
src/coreclr/inc/jiteeversionguid.h	Bumps JIT↔EE version GUID due to interface change.
src/coreclr/inc/icorjitinfoimpl_generated.h	Adds the new virtual method to the generated ICorJitInfo impl surface.
src/coreclr/inc/corinfo.h	Adds the new ICorStaticInfo virtual method for helper resolution.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated no new comments.

[AndyAyersMS]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?
• Could the embedded awaiters hold onto GC refs longer than before?

[jtschuster]

Just one question about test coverage, otherwise crossgen2 changes lgtm.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tests/async/custom-struct-awaiters/custom-struct-awaiters.cs:19

• This xUnit test blocks on an async method via Run().Wait(). xUnit supports async test methods directly, and blocking waits can introduce deadlocks when a synchronization context is present (or hide hangs as AggregateException). Prefer returning Task from the [Fact] instead of blocking.

[Copilot]

Pull request overview

Copilot reviewed 74 out of 74 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:117

• Same as the safe path: this reads TAwaiter from raw continuation data via Unsafe.As, which can generate misaligned struct loads if the awaiter type has alignment requirements beyond pointer-size. Prefer Unsafe.ReadUnaligned<TAwaiter> here as well.
  src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.cs:60
• Unsafe.As<byte, TAwaiter> assumes the source address is suitably aligned for TAwaiter. Continuation member offsets are only guaranteed to be pointer-aligned, so on architectures with stricter alignment requirements this can result in misaligned struct loads. Use Unsafe.ReadUnaligned<TAwaiter> to safely read the awaiter from raw object data.
  src/coreclr/jit/async.cpp:2475
• Stores into the continuation member don't currently propagate GTF_IND_UNALIGNED when the awaiter type's alignment exceeds the object/pointer alignment. Other continuation stores compute indirFlags based on required vs heap alignment; this path should do the same to avoid generating aligned stores to potentially unaligned offsets (especially for SIMD-containing structs).

 GenTreeFieldList* fieldList = awaiter->AsFieldList();
for (GenTreeFieldList::Use& use : fieldList->Uses())
{
if (!use.GetNode()->IsInvariant() && !use.GetNode()->OperIs(GT_LCL_VAR))
{
LIR::Use lirUse(LIR::AsRange(callBlock), &use.NodeRef(), fieldList);
lirUse.ReplaceWithLclVar(m_compiler);

src/coreclr/jit/async.cpp:2506

• This struct store uses GTF_IND_NONFAULTING but omits GTF_IND_UNALIGNED when the awaiter layout has stricter alignment than the continuation/object can guarantee. Mirror the existing pattern used for locals/return stores by computing indirFlags and passing it to gtNewStoreValueNode.

 GenTree* continuation = m_compiler->gtNewLclvNode(GetNewContinuationVar(), TYP_REF);
unsigned offset = OFFSETOF__CORINFO_Continuation__data + layout.ContinuationMemberOffsets[memberIndex];
GenTree* offsetNode = m_compiler->gtNewIconNode((ssize_t)offset, TYP_I_IMPL);
GenTree* address = m_compiler->gtNewOperNode(GT_ADD, TYP_BYREF, continuation, offsetNode);
GenTree* store = m_compiler->gtNewStoreValueNode(awaiterLayout, address, awaiter, GTF_IND_NONFAULTING);
LIR::AsRange(suspendBB).InsertAtEnd(LIR::SeqTree(m_compiler, store));

[Copilot]

Pull request overview

Copilot reviewed 76 out of 76 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/coreclr/jit/async.h:7

• async.h defines non-inline types (ContinuationMemberType, ContinuationMember, etc.) but still has no include guard / #pragma once. Now that more translation units include this header (e.g., gentree.cpp, importercalls.cpp), an accidental double-include in any TU would become a hard compile error (redefinition). Adding a conventional include guard would make this robust.

enum class ContinuationMemberType
{
CustomAwaiterOfLayout,
};

[jakobbotsch]

JIT changes LGTM with few nits.

Some questions:

• Seems like we are now paying up-front for space that we might not ever use. Is there any limit or tradeoff we should consider? Maybe especially so for shared continuations?

• Could the embedded awaiters hold onto GC refs longer than before?

I think it is unlikely that we grow the continuation size by a lot here since the awaiter storage is shared across await sites and is keyed on the awaiter layout. I don't think there will be that many different awaiter layouts in the same method, even with lots of different await sites.

For the GC question: the C# emitted code for the helper that is getting optimized looks like this:

TAwaiterawaiter=somethingAwaitable.GetAwaiter();if(!awaiter.IsCompleted){AsyncHelpers.UnsafeAwaitAwaiter(awaiter);}TResult=awaiter.GetResult();

So the GC refs would already be retained by this.

It also means that there is actually already the necessary state to reconstruct the awaiter stored in the continuation, so we are being a bit wasteful not reusing that. But that is quite complicated and error prone to do:

1. With promotion and optimizations we cannot really guarantee that the state in the continuation is an exact TAwaiter, and I don't see a non-error prone way to guarantee that
2. Without 1 I think we need something like Investigate generating resumption stubs as funclet-like code in runtime async functions #121013 where the JIT generates some funclet, but even if we could generate such funclets I also don't see how we would easily represent this.

So I went with this fix for .NET 11, mostly from worrying about runtime async boxing in cases where async1 wouldn't. I would like to eventually implement 2, but it is definitely going to take some work.

[VSadov]

LGTM!

[jakobbotsch]

Failure in run was #131330

[jakobbotsch]

/ba-g Failure in run was #131330