AccessViolationException with PublishSingleFile

[IgorMilavec]

Description

When publishing a net6.0 application using custom EventListener as as single file, the app crashes with:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
 at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
 at System.Threading.Tasks.Task.InnerInvoke()
 at System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object)
 at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
 at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
 at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
 at System.Threading.Thread.StartCallback()

Reproduction Steps

Program.cs

using System.Diagnostics.Tracing;

EventListener eventListener = new MyEventListener();

Console.WriteLine("Press any key to abort...");

// Do some stuff
for (int i = 1; i < int.MaxValue; i++)
{
 if (Console.KeyAvailable)
 {
 break;
 }

 string a = new('a', i % 10000);
}

Console.WriteLine("Done.");

internal class MyEventListener : EventListener
{
 protected override void OnEventSourceCreated(EventSource eventSource)
 {
 if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime", StringComparison.Ordinal))
 {
 EnableEvents(eventSource, EventLevel.Informational, (EventKeywords)1);
 }
 }

 protected override void OnEventWritten(EventWrittenEventArgs eventData)
 {
 Console.WriteLine($"Event written {eventData.EventId,3} {eventData.EventName ?? string.Empty}");
 }
}

FolderProfile.pubxml:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <PropertyGroup>
 <Configuration>Release</Configuration>
 <Platform>Any CPU</Platform>
 <PublishDir>bin\Release\net6.0\publish\win-x64\</PublishDir>
 <PublishProtocol>FileSystem</PublishProtocol>
 <TargetFramework>net6.0</TargetFramework>
 <RuntimeIdentifier>win-x64</RuntimeIdentifier>
 <SelfContained>true</SelfContained>
 <PublishSingleFile>True</PublishSingleFile>
 <PublishReadyToRun>False</PublishReadyToRun>
 <PublishTrimmed>False</PublishTrimmed>
 </PropertyGroup>
</Project>

Expected behavior

No exceptions, please. :)

Actual behavior

Exception, as described in the Description section.

Regression?

The same app worked in net5.0.

Known Workarounds

Don't use PublishSingleFile. :(

Configuration

.NET SDK (reflecting any global.json):
Version: 6.0.100
Commit: 9e8b04bbff

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
Version: 6.0.0
Commit: 4822e3c

Other information

No response

Tagging subscribers to this area: @agocke, @vitek-karas, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When publishing a net6.0 application using custom EventListener as as single file, the app crashes with:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
 at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
 at System.Threading.Tasks.Task.InnerInvoke()
 at System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object)
 at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
 at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
 at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
 at System.Threading.Thread.StartCallback()

Reproduction Steps

Program.cs

using System.Diagnostics.Tracing;

EventListener eventListener = new MyEventListener();

Console.WriteLine("Press any key to abort...");

// Do some stuff
for (int i = 1; i < int.MaxValue; i++)
{
 if (Console.KeyAvailable)
 {
 break;
 }

 string a = new('a', i % 10000);
}

Console.WriteLine("Done.");

internal class MyEventListener : EventListener
{
 protected override void OnEventSourceCreated(EventSource eventSource)
 {
 if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime", StringComparison.Ordinal))
 {
 EnableEvents(eventSource, EventLevel.Informational, (EventKeywords)1);
 }
 }

 protected override void OnEventWritten(EventWrittenEventArgs eventData)
 {
 Console.WriteLine($"Event written {eventData.EventId,3} {eventData.EventName ?? string.Empty}");
 }
}

FolderProfile.pubxml:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <PropertyGroup>
 <Configuration>Release</Configuration>
 <Platform>Any CPU</Platform>
 <PublishDir>bin\Release\net6.0\publish\win-x64\</PublishDir>
 <PublishProtocol>FileSystem</PublishProtocol>
 <TargetFramework>net6.0</TargetFramework>
 <RuntimeIdentifier>win-x64</RuntimeIdentifier>
 <SelfContained>true</SelfContained>
 <PublishSingleFile>True</PublishSingleFile>
 <PublishReadyToRun>False</PublishReadyToRun>
 <PublishTrimmed>False</PublishTrimmed>
 </PropertyGroup>
</Project>

Expected behavior

No exceptions, please. :)

Actual behavior

Exception, as described in the Description section.

Regression?

The same app worked in net5.0.

Known Workarounds

Don't use PublishSingleFile. :(

Configuration

.NET SDK (reflecting any global.json):
Version: 6.0.100
Commit: 9e8b04bbff

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
Version: 6.0.0
Commit: 4822e3c

Other information

No response

Author:	IgorMilavec
Assignees:	-
Labels:	area-Single-File, untriaged
Milestone:	-

[VSadov]

I am able to repro the issue - EventPipePayloadDecoder.DecodePayload crashes in the context of a singlefile app.

[VSadov]

In my case the app is able to report a few events before crashing.

Press any key to abort...
Event written 9 GCSuspendEEBegin_V1
Event written 8 GCSuspendEEEnd_V1
Event written 35 GCTriggered
Event written 1 GCStart_V2
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 202 GCMarkWithType
Event written 205 GCGlobalHeapHistory_V4
Event written 204 GCPerHeapHistory_V3
Event written 2 GCEnd_V1
Event written 4 GCHeapStats_V2
Event written 7 GCRestartEEBegin_V1
Event written 3 GCRestartEEEnd_V1
Event written 14 GCFinalizersBegin_V1
Event written 13 GCFinalizersEnd_V1
Event written 14 GCFinalizersBegin_V1
Event written 13 GCFinalizersEnd_V1
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
 at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan`1<Byte>)
 at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
 at System.Threading.Tasks.Task.InnerInvoke()
 at System.Threading.Tasks.Task+<>c.<.cctor>b__271_0(System.Object)
 at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
 at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
 at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
 at System.Threading.Thread.StartCallback()

[VSadov]

So far it does not look like this is an issue with missing dependencies, since the app is able to run for a while.
There must be something happening in EventPipePayloadDecoder.DecodePayload that is not compatible with sinlgefile.

@tommcdon @hoyosjs - any guesses what could cause the problem?

[hoyosjs]

My first guess would be some piece of unprotected memory. That should apply to both single file and non-sf though. The other possibility is we have a GC hole or some odd optimization that ends up in this. In single file images are stored as is, right? No trimming or inlining done. The easiest way to get this is run it under WinDBG with time travel enabled and see the address and who assigned to it.

[VSadov]

My repro was without trimming, just a self-contained + singlefile. Without singlefile, just self-contained seems to work fine.

I am not sure how singlefile would expose a GC hole that does not happen in just self-contained. Everything is possible, but sf specific GC hole seems unlikely.
The main difference for singlefile is that the native runtime is the part of the exe. If something needs to get to the native code/data while doing DecodePayload, it could be affected. The native code could also be "trimmed" by the native linker, if it thinks noone calls it.

Yes, will need to debug further. I was just wondering if there are any areas that could be more interesting than others.
TTD seems a good idea, since failure happens fairly quick. Thanks!

[vitek-karas]

I played with this a little:

• Doesn't repro on Debug/Checked build of the runtime
• Reproes with locally built Release runtime (main from dotnet/runtime)

Got this callstack from the debugger:

0:008> !dumpstack
OS Thread Id: 0xaccc (8)
Current frame: (MethodDesc 00007ff67b1a9048 + 0x168 System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan`1<Byte>))
Child-SP RetAddr Caller, Callee
0000003E61D7F220 00007ff6da817c2b singlefilehost!EventPipeInternal_GetNextEvent + 0x5b [F:\dotnet\runtime\src\coreclr\vm\eventpipeinternal.cpp:241], calling singlefilehost!buffer_manager_move_next_event_any_thread [F:\dotnet\runtime\src\native\eventpipe\ep-buffer-manager.c:586]
0000003E61D7F260 00000001803dcff0 (MethodDesc 00007ff67b114fa8 + 0x50 System.Diagnostics.Tracing.EventPipeInternal.GetNextEvent(UInt64, System.Diagnostics.Tracing.EventPipeEventInstanceData*)), calling singlefilehost!JIT_PInvokeEnd
0000003E61D7F2C0 00000001803dd920 (MethodDesc 00007ff67b113cc8 + 0x90 System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan`1<Byte>)), calling CLRDLL: No CLR image loaded (i.e. mscorwks.dll)
00007ff67b04f014 (stub for System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan`1))

[ShaharPrishMSFT]

Hitting this as well in our deployments.

[grahambunce]

hitting this too. Not aware of a custom event listener but I'm using the OpenTelemetry/NLog stack so likely something in there.

.net 6.0.101. self-contained, win-x64, single-file, release or debug mode crashes within a few seconds on start up. Self-contained, win-x64, release mode does not.

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan1)
at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan1<Byte>) at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object) at System.Threading.Thread.StartCallback()