[Flaky test] SslTest.HttpsShouldWork fails on external server errors (504 Gateway Time-out)

[simonrozsival]

Summary

System.NetTests.SslTest.HttpsShouldWork intermittently fails in on-device test runs when the live external endpoints it depends on return an HTTP server error. Most recently it failed with a 504 Gateway Time-out.

Failure

failed HttpsShouldWork
 System.Net.WebException : net_servererror, 504, Gateway Time-out
 at System.NetTests.SslTest.RunIgnoringWebException(Action test)
 at System.NetTests.SslTest.HttpsShouldWork()
 at System.RuntimeMethodHandle.InvokeMethod(...)
 ...
 from /Users/runner/work/1/s/bin/TestRelease/net11.0-android/Mono.Android.NET-Tests.dll (net11.0|x64)

• Build: dnceng-public build 1506492
• Test: System.NetTests.SslTest.HttpsShouldWork
• File: tests/Mono.Android-Tests/Mono.Android-Tests/System.Net/SslTest.cs

Root cause

HttpsShouldWork makes real HTTPS requests to live external URLs:

• https://dotnet.microsoft.com/
• https://www.bing.com/

The call is wrapped in RunIgnoringWebException, which only ignores transient network failures:

case WebExceptionStatus.ConnectFailure:
case WebExceptionStatus.NameResolutionFailure:
case WebExceptionStatus.Timeout:
 return true;

A 504 Gateway Time-out is a server-side HTTP status error (net_servererror, i.e. WebExceptionStatus.ProtocolError), which is not in the ignore list, so the test fails instead of being ignored. The existing per-URL fallback only handles 429 Too Many Requests, not 5xx responses:

if (response.StatusCode == HttpStatusCode.TooManyRequests) {
 // try the next url.
 continue;
}

This is a recurring class of flakiness for this test: it depends on the availability and health of external endpoints that the CI environment does not control.

Previous related issues

• CoreCLR on-device tests #10069 — CoreCLR on-device tests (CLOSED). Referenced by the // TODO: https://github.com/dotnet/android/issues/10069 comment at the top of SslTest.cs; about enabling CoreCLR on-device tests generally, not this specific flakiness.
• [TrimmableTypeMap] Resolve all TrimmableIgnore, SSL, and Java.Interop-Tests exclusions #11099 — [TrimmableTypeMap] Resolve all TrimmableIgnore, SSL, and Java.Interop-Tests exclusions (CLOSED). Touches SSL test exclusions, not flakiness.

No existing issue tracks this specific failure.

Previous related PRs (network-resilience history for this test)

• [tests] More network resilience for tests #2954 — [tests] More network resilience for tests (2019, merged). Added the RunIgnoringWebException / ShouldIgnoreException helper that ignores ConnectFailure / NameResolutionFailure / Timeout. This is why the current 504 slips through — it is not one of those statuses.
• 112c8328a6 (2023) — Added the multi-URL list (dotnet.microsoft.com, bing.com) plus the HttpStatusCode.TooManyRequests (429) fallback. Handles 429, not 5xx.
• [tests] Mark ProxyTest as Inconclusive on network failure #11288 — [tests] Mark ProxyTest as Inconclusive on network failure (2026-05, merged). Applied the same resilience pattern to ProxyTest, explicitly modeled on SslTest.
• [tests] Use local servers for networking tests #12022 — [tests] Use local servers for networking tests (2026-07, merged). Migrated sibling SSL/net tests to local loopback / self-signed HTTPS servers, but HttpsShouldWork was left on live external URLs (it only removed httpbin.org from the list). This test is the remaining external-URL dependency.

Suggested fix

Either of:

1. Preferred: Convert HttpsShouldWork to use a local loopback / self-signed HTTPS server, consistent with the direction of [tests] Use local servers for networking tests #12022.
2. Minimal: Treat HTTP 5xx server errors (e.g. 504) as ignorable / inconclusive network failures, matching the existing 429 handling and RunIgnoringWebException behavior.