Fix Http2 MultiConnection test race conditions by MihaZupan · Pull Request #91156 · dotnet/runtime

MihaZupan · 2023-08-26T13:04:14Z

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

MihaZupan · 2023-08-26T13:04:14Z

Closes#91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

ghost · 2023-08-26T13:04:27Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Closes #91075

These tests have logic along the lines of

varconnection1=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();varconnection2=awaitSetupConnectionAsync(StreamLimit);awaitStartStreamLimitRequestsAsync();

When a new Http2 connection is being added to the pool, it will first check whether there are any pending requests on the queue. It will signal those requests before storing itself in the list of available connections on the pool.

When the warmup request in SetupConnectionAsync was woken up, it would be fully processed quickly. We would then continue to queue up new client requests assuming we already have a connection in the pool to handle them.
However due to the race condition, the connection hasn't added itself to the pool yet, so these queued requests end up kicking off a new connection attempt.

You can then get into a situation where the next SetupConnectionAsync will receive a request, but it won't be the same request as the warmup task. Because the warmup task never got a response, the test would time out.

This change fixes this race by making sure to accept all pending requests on the server first, before initiating a new connection. As all previous requests will be accepted already, SetupConnectionAsync will see the first request on the connection being the warmup one correctly.

The issue happening with Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated is a slight derivation of the above.
We would also start an extra connection attempt, but as the test involves waiting 20 seconds for the connection to hit the pool idle timeout, that connection attempt would end up hitting PendingConnectionTimeoutOnRequestCompletion.
When we went to accept a new connection after the fact, the loopback server would accept that already-cancelled connect attempt, and fail when trying to use that connection.

This change fixes that by blocking that third connection attempt from going through ConnectCallback to the socket layer until we're ready to accept it.

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	`area-System.Net.Http`
Milestone:	9.0.0

MihaZupan · 2023-08-26T13:10:15Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2023-08-26T13:10:23Z

Azure Pipelines successfully started running 1 pipeline(s).

wfurt

LGTM

MihaZupan · 2023-08-30T16:04:35Z

/backport to release/8.0

github-actions · 2023-08-30T16:04:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6027293964

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Fix Http2 MultiConnection test race conditions
84371ea

MihaZupan added the area-System.Net.Http label Aug 26, 2023

MihaZupan added this to the 9.0.0 milestone Aug 26, 2023

MihaZupan requested a review from a teamAugust 26, 2023 13:04

MihaZupan self-assigned this Aug 26, 2023

wfurt approved these changes Aug 28, 2023
View reviewed changes

MihaZupan merged commit 36cf0e4 into dotnet:mainAug 29, 2023

github-actionsBot mentioned this pull request Aug 30, 2023
[release/8.0] Fix Http2 MultiConnection test race conditions #91343
Merged

karelz mentioned this pull request Aug 31, 2023
Http2_MultipleConnectionsEnabled test failures #91075
Closed

ghost locked as resolved and limited conversation to collaborators Sep 29, 2023

antonfirsov added the test-bug Problem in test source code (most likely) label Dec 12, 2024

Uh oh!

Conversation

MihaZupan commented Aug 26, 2023

Uh oh!

ghost commented Aug 26, 2023

Uh oh!

MihaZupan commented Aug 26, 2023

Uh oh!

azure-pipelinesBot commented Aug 26, 2023

Uh oh!

wfurt left a comment

Choose a reason for hiding this comment

Uh oh!

MihaZupan commented Aug 30, 2023

Uh oh!

github-actionsBot commented Aug 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MihaZupan commented Aug 26, 2023

Uh oh!

ghost commented Aug 26, 2023

Uh oh!

MihaZupan commented Aug 26, 2023

Uh oh!

azure-pipelinesBot commented Aug 26, 2023

Uh oh!

wfurt left a comment

Choose a reason for hiding this comment

Uh oh!

MihaZupan commented Aug 30, 2023

Uh oh!

github-actionsBot commented Aug 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MihaZupan commented Aug 26, 2023

Uh oh!

ghost commented Aug 26, 2023

Uh oh!

MihaZupan commented Aug 26, 2023

Uh oh!

azure-pipelinesBot commented Aug 26, 2023

Uh oh!

wfurt left a comment

Choose a reason for hiding this comment

Uh oh!

MihaZupan commented Aug 30, 2023

Uh oh!

github-actionsBot commented Aug 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MihaZupan commented Aug 26, 2023

Uh oh!

ghost commented Aug 26, 2023