Exclude runtime-async methods from built-in COM vtables and GUID calculations

[jkoritzinsky]

Adjust built-in COM to interact correctly with runtime-async.

For CCWs, async-call-conv methods are excluded from the vtable and CLSID/IID computation. This ensures that auto-generated IIDs/CLSIDs for the IClassX scenario are unchanged. Additionally, this ensures that the Task-returning (compiler-async call conv) variants are the variants present in the vtable, preserving existing behavior.

For RCWs, the async-call-conv methods are excluded from the vtable slot calculation and the async thunk variants are adjusted to not be treated as CLR->COM methods and instead call the Task-returning thunks on the ComImport interface (similar to how they behave regularly).

Also adds tests that run with both compiler async and runtime async to ensure equivalent behavior.

Fixes#121765

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR ensures that built-in COM interop correctly handles runtime-async methods by excluding them from vtable and GUID calculations. For CCWs, async-call-conv methods are omitted from the vtable and CLSID/IID computation to preserve backward compatibility with existing IClassX IIDs/CLSIDs. For RCWs, async-call-conv methods are excluded from vtable slot calculations, and async thunks are adjusted to call Task-returning interface methods using CALLVIRT.

Key changes:

• Excludes runtime-async methods from COM vtable and interface ID generation
• Adds SparseVTableMap::RecordExcludedMethod to track excluded async methods
• Updates async thunk generation to use CALLVIRT for abstract ComImport methods
• Adds comprehensive tests validating vtable layout for both compiler-async and runtime-async scenarios

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/coreclr/vm/methodtablebuilder.cpp	Records async variant methods as excluded in COM vtables when building method tables
src/coreclr/vm/method.cpp	Adds assertion that async methods don't request COM slots
src/coreclr/vm/interoputil.cpp	Converts ThrowHR to assertion for async methods in stringified interface definitions
src/coreclr/vm/dispatchinfo.cpp	Returns NULL instead of throwing for runtime-async methods in IDispatch scenarios
src/coreclr/vm/comtoclrcall.cpp	Removes redundant async method check and adds precondition assertions
src/coreclr/vm/commtmemberinfomap.cpp	Skips async methods when building COM member info maps and handles null entries
src/coreclr/vm/comcallablewrapper.cpp	Filters out async methods during CCW vtable layout for both classes and interfaces
src/coreclr/vm/clrtocomcall.cpp	Fixes switch statement formatting and confirms async methods not supported on IDispatch
src/coreclr/vm/class.h	Declares RecordExcludedMethod for SparseVTableMap
src/coreclr/vm/class.cpp	Implements RecordExcludedMethod to track excluded vtable slots
src/coreclr/vm/asyncthunks.cpp	Uses CALLVIRT for abstract task-returning variants (ComImport scenarios)
src/tests/Interop/COM/ServerContracts/Server.Contracts.h	Adds COM interface definitions with runtime-generated IIDs for test validation
src/tests/Interop/COM/RuntimeAsync/RuntimeAsync.cs	Tests that async methods don't modify CCW/RCW vtables
src/tests/Interop/COM/RuntimeAsync/TaskComServer.cs	Implements COM server with Task-returning methods for testing
src/tests/Interop/COM/RuntimeAsync/RuntimeAsyncNative.cpp	Native functions to validate vtable slot layout from C++
src/tests/Interop/COM/RuntimeAsync/RuntimeAsync.csproj	Test project with runtime-async feature enabled
src/tests/Interop/COM/RuntimeAsync/CompilerAsync.csproj	Test project using compiler async (baseline comparison)
src/tests/Interop/COM/RuntimeAsync/CMakeLists.txt	Build configuration for native test library
src/tests/Interop/CMakeLists.txt	Adds RuntimeAsync subdirectory to build

[AaronRobinsonMSFT]

Should we also update the COM source generator to avoid generating these signatures? Basically any interface marked with GeneratedComInterface can't have an async method?

[jkoritzinsky]

Should we also update the COM source generator to avoid generating these signatures? Basically any interface marked with GeneratedComInterface can't have an async method?

I don't think we need to block anything for GeneratedComInterface, as using [return:MarshalUsing(typeof(Marshaller))] with an async Task or async Task<T> is entirely valid.

Unlike built-in COM, we don't need to worry about the runtime introducing additional synthetic members as we burn in vtable offsets at compile time (and even if we set them up at runtime, the async variants are hidden from reflection so we wouldn't accidentally see them).

[jkotas]

When is this encountered? Do we have a test for this?

[jkoritzinsky]

I hit this in the RCW side of the tests I added.

Without this, the async variant tries to call the Task-returning CLR->COM interface method with the call instruction instead of callvirt, with results in an exception from the JIT for invalid IL.

[jkotas]

LGTM. Thank you for adding the tests