Use SOS to dump managed stack traces from a dump on Windows

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.

[jkoritzinsky]

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

[ghost]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Download SOS as part of our Helix jobs and load it into cdb to dump managed stack traces when a test crashes or times out.

Also introduces a synthetic test failures to validate behavior (will be removed before merging).

Author:	jkoritzinsky
Assignees:	jkoritzinsky
Labels:	area-Infrastructure
Milestone:	-

[stephentoub]

Before dotnet-sos existed, we used clrmd directly in RemoteExecutor:
https://github.com/dotnet/arcade/blob/bdc59254cf108e1d48451dc43bb9ebc331cdca7b/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs#L177-L225
We might want to standardize on one or the other.

[jkoritzinsky]

ClrMD there is doing process attach only for timeouts, whereas we're handling both timeouts and crashes here, so it's a little different. Definitely worth considering consolidation though.

[jkoritzinsky]

There were some talks about standardizing this sort of mechanism (dumping stack traces from a dump to stdout for the build analysis tooling), but with the recent changes to the EngSrv teams, I don't know how much of that work will still happen.

[stephentoub]

Right, I understand they're handling different cases. But they can both handle both, so it's overkill to have two different tools used for the same purpose. Simply suggesting we choose one and stick with it.

[jkoritzinsky]

I'll look at the RemoteExecutor implementation and see if we can consolidate something useful.

[stephentoub]

Cool, thanks.

[jkoritzinsky]

We'll need to use something like this pattern to get both native and managed stack traces, and we'll likely want to use this model as well to solve #83047 (which focuses more on crashes than timeouts) if we don't move to using dotnet test for the libraries tests in Helix. The RemoteExecutor version has a much better UX around the output though as it has more structure to work with (from CLRMD's APIs).

[jkoritzinsky]

Marking this ready for review as the change works (see the logs for the CoreCLR windows runtime tests), but marked as no-merge as I need to remove the induced test failure.

@hoyosjs for review.

[hoyosjs]

@jkoritzinsky this looks good now. It's not loading the PDBs for managed for for line numbers, but that's separate.

[jkoritzinsky]

Networking failures are known issues, wasm timeout is unrelated. Merging this in. I'll file an issue to follow-up on providing a consistent story for dumping stacks on crashes.