Fix failing test on NativeAOT by noahfalk · Pull Request #109853 · dotnet/runtime

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:11:49Z

Fixes#109828
This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-15T01:12:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T01:12:25Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-15T23:15:39Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-15T23:15:55Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-16T23:13:49Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-16T23:13:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk · 2024-11-17T01:00:28Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-11-17T01:00:57Z

Azure Pipelines successfully started running 1 pipeline(s).

noahfalk · 2024-11-18T22:40:56Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

MichalStrehovsky · 2024-11-18T22:51:26Z

@jkotas @MichalStrehovsky - I think this resolves the failure in the allocation sampling test though it appears the outer loop runs still have some other failures in the Numerics tests.

That one was fixed in #109842. We could rebase and recheck, but I think this is good to merge as-is! Thanks!

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

tommcdon

Thanks!

noahfalk · 2024-11-19T00:17:23Z

Btw, we are able to compute type names of everything on the GC heap (since we keep it around for object.GetType to work). If it's possible to call into managed code at the spot where this is needed (we can only compute the names in managed code), it should be fixable.

Ah thats good to know. I had been under the impression the names were strippable like other metadata and couldn't be assured. The place where we need the name is here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/Runtime/GCHelpers.cpp#L485

The callstack would look like:

FireAllocationSampled
GcAllocInternal
RhAllocateXYZ
ManagedCode

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion. Not knowing what is involved just yet, do you think we could keep that call allocation-free?

jkotas · 2024-11-19T02:59:36Z

I assume there isn't anything preventing a native->managed call at that point but it would be a little odd if managed code did any allocations which could lead to recursion.

Right, it would not be pretty to make this work.

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

noahfalk · 2024-11-19T23:59:40Z

Sending the type name as part of each sample can result in a lot of redundant information being transferred. Would it be better to send the type id to type name mapping in separate events, once for each type id? It would work better for native AOT as well.

It is possible to create separate events that do id->name mapping, but that bring some alternate complexity to track which mappings need to be sent and dealing with potential dropped events carrying the mapping data. Historically all the AllocationTick events carried the name information inline and I'm not aware of any complaints about data size. The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already. If we get some feedback from profiler vendors that ID->name mapping events would be helpful nothing precludes us from adding it (as well as offering a name-free variant of the AllocationSampled event) but I'm going to hold off on increasing the feature scope for now.

MichalStrehovsky · 2024-11-20T06:03:16Z

The plan for NativeAOT is that TypeId can be looked up in a PDB. Assuming the profiler cares about stack traces where the allocations are occurring it will need to do PDB lookups for code IPs already.

The stack traces are a similar story - we do have that information (unless the user specified StackTraceSupport=false property), but it's only computable in managed code.

The information that we have in metadata both for types and method bodies is more structured than in the PDB - the PDB only has mangled names that are not particularly reversible. It works, but it won't produce nice identifier names. But it's better than nothing, and good enough 98% of time.

I guess none of this is something that we would need to address now, just something to keep in mind should we have a need for this.

Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

ghost added the area-Tracing-coreclr label Nov 15, 2024

dotnet-policy-serviceBot assigned noahfalk Nov 15, 2024

build-analysisBot mentioned this pull request Nov 15, 2024
The hosted runner encountered an error while running your job. (Error Type: Disconnect). dotnet/dnceng#1919
Closed
3 tasks

noahfalk force-pushed the fix_alloc_test branch from 96916a6 to 0b0cab3Compare November 15, 2024 11:06

noahfalk force-pushed the fix_alloc_test branch from 0b0cab3 to cb13c1bCompare November 16, 2024 23:12

Fix failing test on NativeAOT
a614c2b
Fixesdotnet#109828 This test hadn't been updated to account for NativeAOT's lack of type names in the new randomized sampling allocation events.

noahfalk force-pushed the fix_alloc_test branch from cb13c1b to a614c2bCompare November 17, 2024 01:00

tommcdon approved these changes Nov 18, 2024
View reviewed changes

noahfalk merged commit e33be4d into dotnet:mainNov 19, 2024

github-actionsBot locked and limited conversation to collaborators Dec 20, 2024