Fix thread pool hang

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.

[kouvel]

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that there are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes#55642

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• In Improve the rate of thread injection for blocking due to sync-over-async #53471 the thread count goal was moved out of ThreadCounts, it turns out that are a few subtle races that it was avoiding. There are other ways to fix it, but I've added the goal back into ThreadCounts for now.
• Reverted PR mitigation for quic tests hangs #55985, which worked around the issue in the CI

Fixes #55642

Author:	kouvel
Assignees:	kouvel
Labels:	area-System.Threading
Milestone:	6.0.0

[stephentoub]

Thanks for tracking this down!

There was some suspicion this may also have been contributing to some of the CI hangs we've seen recently. It'll be interesting to see if their rate decreases after this goes in.

[danmoseley]

Is there anything that could be done to create a test that would catch this kind of failure? Or do we already have that in the Quic tests 😄

[wfurt]

Seems like tricky business @danmoseley. It was found by standard conformance Stream tests (with synchronous IO).
On 2Core machine and sync-over-async inside MsQuic wrapper.

[wfurt]

The QUIC part looks good to me. I'll give it try tomorrow.

[danmoseley]

True, but I have seen @kouvel devise unit tests that consistently repro the most sporadic issues, so I thought I'd ask 😄

[davidfowl]

I wonder if coyote could come in handy here https://microsoft.github.io/coyote/

[kouvel]

I had some difficulty in reproing it initially when I tried but upon trying a few more things I got a fairly frequent repro now on my Windows machine, added a test.