Uh oh!
There was an error while loading. Please reload this page.
Disable core dumps when running libraries tests on macOS - #65405
Conversation
ghost
commented
Feb 15, 2022
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries Issue Details@dotnet/area-infrastructure-libraries
|
| # See discussions in: | ||
| # https://github.com/dotnet/core-eng/issues/15333 | ||
| # https://github.com/dotnet/core-eng/issues/15597 | ||
| ulimit -c 0 |
There was a problem hiding this comment.
This will disable system cores - are there any places where we hook up the runtimes coredump features like we do for the coreclr style tests? I am not aware of any, and they shouldn't be as bad in terms of size. Those could let us still get dumps reasonably.
There was a problem hiding this comment.
I did not find that for libraries tests.
I did query for dump uploads from runtime/non-libraries tests which I believe use createdump and they still look in the 6 GB range?
WorkItems
| where QueueName startswith"osx.1200.amd64"
| where Started > ago(10d)
| join (Files | where FileName startswith"core" ) on WorkItemId
| joinkind=inner Jobs on JobId
| project
Queued,
WorkItemFriendlyName, ExitCode, QueueName,
PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
Pipeline = tostring(parse_json(Properties).DefinitionName),
FileName,
SizeBytesLong,
Source,
Build = tostring(parse_json(Properties).BuildNumber)
| where Pipeline == "runtime"
| where PhaseName !startswith("libraries")
| orderby Queued descThere was a problem hiding this comment.
There's two things: we should remove this https://github.com/dotnet/runtime/blob/512a9ffcb8aa99ffc25fac76dc0c431543e45ba9/src/tests/Common/Coreclr.TestWrapper/CoreclrTestWrapperLib.cs#L223 and yeah, we are using --with-heap. That's pretty much grab everything in the managed process. It gives us the best results, but if the size is too large maybe we should consider dropping to --normal or --triage to at least get stacks.
There was a problem hiding this comment.
It would be great if the libraries tests could automatically wrap in coredump. I can't remember ever needing a dump from any process other than the test process (or possibly a child). So we could leave system dumps off at that point.
There was a problem hiding this comment.
Also update here and set DbgMiniDumpType, I expect:
runtime/src/tests/Common/testenvironment.proj
Lines 79 to 81 in 03494f3
danmoseley
commented
Feb 16, 2022
We're expecting to enable this again pretty soon - matter of days, hopefully? Just waiting on @ChadNedzlek to put in some throttling? |
elinor-fung
commented
Feb 16, 2022
Even with the throttling, we would still have caused a lot of stress - it would have self-resolved eventually, but since the number of work items is more than the number of machines, a single PR causing wide-spread crashes could take out the entire queue for hours (there was a solid First Responders thread on this). |
We had a single PR - with a real bug crashing all the tests - take out the whole queue. Disabling core dumps for test runs on macOS entirely until we have some better story.
cc @dotnet/area-infrastructure-libraries @danmoseley@stephentoub